General Motors uses telematics data as a cornerstone of its software-defined vehicle strategy to enhance safety, security, and product development, aiming to create an exceptional customer experience. The analytical datastore team owns the platform that streams and stores this data from millions of vehicles globally, and this role is focused on building a scalable data platform to support increasing data volume and complexity.
Requirements
- 5+ years of experience in data engineering including Python or Scala, SQL, and relational/non-relational data storage
- 3+ years of experience in distributed, petabyte-scale data processing using Spark as well as container orchestration technologies like Kubernetes
- Strong with performance tuning strategies such as partitioning, clustering, caching, and various other serialization techniques
- Strong with infrastructure as code (IaC) and CI/CD best practices
- Proven cloud experience and familiarity with at least one cloud platform (Azure preferred)
- Strong background in any of the following: data architecture, data governance, metadata management, data quality, observability, or MLOps
- Knowledge in schema design and data contracts
Responsibilities
- Develop a scalable data platform to support continuing increases in data volume and complexity
- Develop and maintain data pipelines to ingest, process, and curate large volumes of data using Spark, Kafka, Event Hubs, Azure Data Lake Storage, or similar technologies
- Support the infrastructure required for optimal ingestion, transformation, and storing of data
- Collaborate with data scientists to support the needs of advanced ML development
- Design and implement rigorous data observability, validation, governance, and quality checks
- Mentor and support junior engineers by providing guidance, coaching and educational opportunities
- Simplify, simplify, simplify… and document your work!
Other
- Bachelor's degree in Computer Science, Software Engineering, or a related field
- Master's degree in Computer Science, Software Engineering, or a related field (preferred)
- This role is categorized as hybrid and requires the successful candidate to report to Austin, TX or Warren, MI three times per week, at minimum
- Excellent teamwork and leadership skills to collaborate and influence across product, program, and engineering
- GM DOES NOT PROVIDE IMMIGRATION-RELATED SPONSORSHIP FOR THIS ROLE
