eSimplicity is looking to hire a Staff Software Engineer to support the ongoing operation and re-architecture of their data ingestion capability, which routinely supports large volumes of Medicare and Medicaid data. The goal is to build scalable, resilient data pipelines that ingest, validate, and transform data rapidly and accurately, emphasizing observability and reliability.
Requirements
- Extensive Data pipeline experience using Python, Java, and cloud technologies
- Expert data pipeline builder and data wrangler who enjoys optimizing data systems and building them from the ground up.
- Experienced in designing data architecture for shared services, scalability, and performance
- Experienced in designing data services including API, metadata, and data catalog.
- Experienced in data governance process to ingest (batch, stream), curate, and share data with upstream and downstream data users.
- Ability to build and optimize data sets, ‘big data’ data pipelines, and architectures?
- Demonstrated understanding and experience using software and tools, including big data tools like Kafka, Spark, and Hadoop; relational NoSQL and SQL databases including Cassndra and Postgres; workflow management and pipeline tools such as Airflow, Luigi and Azkaban; AWS cloud services including Redshift, RDS, EMR, and EC2; stream-processing systems like Spark-Streaming and Storm; and object function/object-oriented scripting languages including Scala, C++, Java, and Python.?
Responsibilities
- Creates new pipeline development and maintains existing pipeline; updates Extract, Transfer, Load (ETL) process; creates new ETL feature development; builds PoCs with Redshift Spectrum, Databricks, etc.;
- Implements, with the support of project data specialists, large dataset engineering: data augmentation, data quality analysis, data analytics (anomalies and trends), data profiling, data algorithms, and (measure/develop) data maturity models and develop data strategy recommendations.
- Assemble large, complex data sets that meet non-functional and functional business requirements.
- Identify, design, and implement internal process improvements, including re-designing data infrastructure for greater scalability, optimizing data delivery, and automating manual processes. ?
- Building required infrastructure for optimal extraction, transformation, and loading of data from various data sources using AWS and SQL technologies
- Building analytical tools to utilize the data pipeline, providing actionable insight into key business performance metrics, including operational efficiency and customer acquisition?
- Write unit and integration tests for all data processing code.
Other
- Leads and mentors all other data roles in the program.
- All candidates must pass public trust clearance through the U.S. Federal Government.
- Minimum of 10 years related experience.
- With ten years of general information technology experience and at least eight years of specialized experience, a degree is NOT required.
- Flexible and willing to accept a change in priorities as necessary.
