New York eHealth Collaborative (NYeC) is looking to improve healthcare by collaboratively leading, connecting, and integrating health information exchange across the State. The Data Engineer will be responsible for managing and optimizing data pipelines and ensuring data quality and fidelity for the Statewide Health Information Network for New York (SHIN-NY).
Requirements
- Proven experience with high-volume data processing and designing solutions for reliability and performance at scale.
- Deep understanding of data architecture, data modeling, and ETL processes.
- Proficiency in SQL and Python; experience with Snowflake, Redshift, Databricks, and other cloud-based data platforms.
- Experience writing AWS Lambda, Airflow, AWS Glue; familiarity with serverless frameworks.
- Knowledge of NoSQL databases, especially Snowflake and DynamoDB Streams.
- Experience with APIs: Postman/API troubleshooting skills, API Gateway, AWS SDKs, OAuth 2.0/OpenID Connect.
- Experience with healthcare standards and technologies such as HL7, CDA, PIXV3, FHIR, HAPI FHIR, e-prescribing, and meaningful use.
Responsibilities
- Write queries for reporting, analysis, and data quality detection.
- Create new fields for ingesting new data or tracking data flow for reporting.
- Leverage existing data models and factor in ways that optimize sharing.
- Develop comprehensive schema designs that effectively address business needs and support data-driven decision-making.
- Minimize compute time of Snowflake queries, identify problem queries, and tune existing queries.
- Build and maintain scalable ETL pipelines between systems.
- Build and maintain AWS Lambda APIs and integrations with DynamoDB and Snowflake.
Other
- Hybrid work schedule.
- Ability to work independently and within a team environment.
- Strong organizational skills with exceptional attention to detail.
- Employees work a hybrid in-office schedule (at either our Manhattan, NY or Albany, NY office and then remotely).
- From time to time our team members must visit other parts of the state.
