GM Financial is looking to solve the problem of building and maintaining reliable and scalable data pipelines for ABS reporting, enabling seamless data access and analysis for accounting business users.
Requirements
- Experience with processing large data sets using Hadoop, HDFS, Spark, Kafka, Flume or similar distributed systems
- Experience with ingesting various source data formats such as JSON, Parquet, SequenceFile, Cloud Databases, MQ, Relational Databases such as Oracle
- Experience with Cloud technologies (such as Azure, AWS, GCP) and native toolsets such as Azure ARM Templates, Hashicorp Terraform, AWS Cloud Formation
- Understanding of cloud computing technologies, business drivers and emerging computing trends
- Working knowledge of Object Storage technologies to include but not limited to Data Lake Storage Gen2, S3, Minio, Ceph, ADLS etc
- Experience with containerization to include but not limited to Dockers, Kubernetes, Spark on Kubernetes, Spark Operator
- Strong background with source control management systems (GIT or Subversion); Build Systems (Maven, Gradle, Webpack); Code Quality (Sonar); Artifact Repository Managers (Artifactory), Continuous Integration/ Continuous Deployment (Azure DevOps)
Responsibilities
- Work internal business partners to identify, capture, collect, and format data from the external sources, internal systems and the data warehouse to extract features of interest
- Contribute to the evaluation, research, experimentation efforts with batch and streaming data engineering technologies in a lab to keep pace with industry innovation
- Work with data engineering related groups to inform on and showcase capabilities of emerging technologies and to enable the adoption of these new technologies and associated techniques
- Create and implement business rules and functional enhancements for data schemas and processes
- Perform data load monitoring and resolution
- Work with internal business clients to problem solve data availability and activation issues
- Coordinate with Privacy Compliance to ensure proper data collection and handling
Other
- 2-4 years of hands on experience with data engineering required
- Bachelor’s Degree in related field or equivalent experience required
- Ability to work in an Agile/Scrum team environment
- Strong interpersonal, verbal, and writing skills
- Minimum of 2-days a week in office in Fort Worth, Texas
