Lead Data Engineer in Westlake Village, CA at Consultnet Careers

Date Posted: 3/8/2023

Job Snapshot

Job Description

Lead Data Engineer (Hybrid)


Westlake Village, CA


Direct Hire


$130k- $165k



In this role you will be expected to provide strong leadership while developing the Enterprise data architecture of Digital and Retail businesses across several different brands. You will be an SME who provides technical guidance, architecture, and a fierce advocate of technical standards. You will provide assessments and prototype new concepts and technologies and often develop capabilities to be handed off to operational development teams. You will provide analysis on complex problems as well as be an active and involved contributor in enterprise technology design decisions.



Responsibilities/Duties


  • Responsible for modernization, transformation and migration to a cloud data platform

  • Design and build reliable and scalable data infrastructure with AWS technologies utilizing leading privacy and security techniques to safeguard data

  • Architect scalable, secure, low latency, resilient and cost-effective solutions for enabling predictive and prescriptive analytics across the organization

  • Design/Architect frameworks to Operationalize Client models through serverless architecture and support unsupervised continuous training models

  • Assume responsibility for and scale the organization's data models (Tableau, Kibana, Dynamo DB)

  • Share data-backed findings to various internal and external stakeholders

  • Build frameworks for data ingestion pipeline both real time and batch using best practices in data modeling, ETL/ELT processes, and hand off to data engineers

  • Complete code reviews audit implementations, and provide constructive feedback to help others create better solutions.

  • Stay abreast of tech industry trends and technologies, actively drive technology direction, and make technological recommendations based on experience and research.


Requirements


  • 10+ years of experience working directly with enterprise data solutions

  • Practical experience working in a public cloud environment and on-prem infrastructure.

  • Provable experience with Columnar Databases like Redshift Spectrum, Time Series data stores like Apache Pinot and the AWS cloud infrastructure

  • Experience with in-memory, serverless, streaming technologies and orchestration tools such as Spark, Kafka, Airflow, Kubernetes

  • Current/Recent hands-on implementation experience is a must, with 7+ years of IT platform implementation experience.




  • AWS Certified Big Data Certification is a plus

  • Experience designing and implementing AWS big data and analytics solutions in large digital and retail environments is a plus

  • Expertise with online transactional (OLTP) processing and analytical processing (OLAP) databases, data lakes, and schemas.

  • Experience with AWS Cloud Data Lake Technologies and operational experience of Kinesis/Kafka, S3, Glue and Athena.

  • Experience with a variety of the message/file formats: Parquet, Avro, ORC

  • Experience in the design and development with Streaming Service, EMS, MQ, Java, XSD, File Adapter, and ESB based applications

  • Experience with distributed architectures such as Microservices, SOA, RESTful APIs and data integration architectures.

  • Recent hands-on experience migrating On-Prem Data solutions to the cloud

  • Experience managing On-prem Enterprise Data Warehouse solutions like Netezza is a plus

  • Experience and Expertise with a variety of modern data processing technologies, including:

    • Big Data Stack (Spark, spectrum, Flume, Kafka, Kinesis etc.)

    • Data streaming (Kafka, SQS/SNS queuing, etc)

    • Columnar databases primarily, Redshift or like technologies lile Snowflake, Firebolt

    • AWS services (S3, Lambda, Redshift, Glue, EC2, etc)

    • Python, pySpark or similar programming languages is a must

    • BI tools (Tableau, Domo, MicroStrategy) is a plus

    • Proficient with AWS Compute such as EC2, Lambda, Beanstalk, or ECS

    • Proficient with AWS Management and Governance suite of products such as CloudTrail, CloudWatch, or Systems Manager

    • Proficient with Amazon Web Services (AWS) offerings, development, and networking platforms

    • Proficient with AWS Analytics such as Athena, EMR, or Glue

    • Proficiency in Oracle, MYSQL and Microsoft SQL Server Databases is a plus

    • Understanding of CI/CD with experience in Jenkins





Be a part of the ConsultNet difference. As a leading national provider of IT staffing and solutions, ConsultNet delivers exceptional services to startup, midmarket and Fortune 1000 companies across North America. Since 1996, we've partnered with clients to create rewarding opportunities for our consultants, successfully building teams that have surefire results.


In the past two years alone, we have placed more than 1,500 consultants in contract, contract-to-hire, or direct placement opportunities. We understand communication is key to finding the right job that matches your skills and career goals. For us, it's not just the work that we do; it's how we do the work. Our breadth of offerings extends to multiple IT positions in major markets throughout the country, see more at - www.consultnet.com