As a Data Engineer, you will build industrialized data assets and optimize data pipelines in support of Business Intelligence and Advance Analytic objectives for General Motors.
Requirements
- 5 + years' experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
- Strong and hands-on experience in databricks data engineering and management.
- Experience with setting up CI/CD environment in Azure and Databricks leveraging Azure Devops, ARM templates, GitHub actions and DBX Asset bundles and Terraform etc.
- Strong understanding of Gen AI tools and proven ability to integrate AI tools such as Microsoft co-pilot, data bricks genie with daily tasks in Databricks.
- Experience with big data frameworks and tools like Apache Hadoop, Apache Spark, or Apache Kafka for processing and analyzing large datasets.
- Strong understanding and ability to provide mentorship in the areas of data ETL processes and tools for designing and managing data pipelines.
- Experience in designing streaming data pipelines using FiveTran, Azure Event Hubs, Auto Loader and delta lake in Azure Databricks.
Responsibilities
- Communicate and maintain Master Data, Metadata, Data Management Repositories, Logical Data Models, Data Standards
- Create and maintain optimal data pipeline architecture
- Assemble large, complex data sets that meet functional / non-functional business requirements
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build industrialized analytic datasets and delivery mechanisms that utilize the data pipeline to deliver actionable insights into customer acquisition, operational efficiency and other key business performance metrics
- Work with business partners on data-related technical issues and develop requirements to support their data infrastructure needs
- Ability to create highly consistent and accurate analytic datasets suitable for business intelligence and data scientist team members
Other
- This role is based remotely but if you live within a 50-mile radius of an office [Atlanta, Austin, Detroit, Warren, or Mountain View], you are expected to report to that office three times a week, at minimum.
- Ability to work effectively in cross-functional teams, collaborate with data scientists, analysts, and stakeholders to deliver data solutions.
- Ability to multi-task and stay organized in a dynamic work environment
- This job is not eligible for relocation benefits. Any relocation costs would be the responsibility of the selected candidate.
- The expected base compensation for this role is: $94,800 - $159,600.
