ECU Health is looking to improve the health and well-being of eastern North Carolina through patient care, education, and research by designing, developing, and maintaining data pipelines and infrastructure to support AI and machine learning applications.
Requirements
- Azure Services: Data Factory, Synapse, Data Lake Gen2, Stream Analytics, Event Hubs, Azure ML, Key Vault, Purview
- Big Data & Processing: Azure Databricks, PySpark, Delta Lake
- Languages: Python, SQL, Scala (optional)
- CI/CD & IaC: Azure DevOps, GitHub Actions, Terraform, Bicep
- Monitoring & Logging: Azure Monitor, Log Analytics, Application Insights
- Governance & Cataloging: Microsoft Purview, Azure Policy
- Proven experience in infrastructure as code (e.g., Terraform, CloudFormation)
- Proven experience in data pipeline development and ETL processes
Responsibilities
- Design and implement ETL processes to extract, transform, and load data from diverse sources into central data storage systems such as data warehouses or data lakes and implement batch and real-time data pipelines using Azure Data Factory, Azure Synapse Pipelines, and Azure Stream Analytics.
- Develop scalable data pipelines that can handle large volumes of data with high velocity and variety within an Epic EHR system.
- Ensure data quality and integrity by implementing data validation and cleansing procedures (e.g., SQL Server, Blob Storage, Event Hubs, IoT Hub, APIs) into Azure Data Lake Gen2 or Synapse Analytics.
- Select and manage appropriate data storage solutions, including SQL, NoSQL, and cloud-based data warehouses using Delta Lake and Parquet formats to enable performant, versioned data storage for ML training.
- Build reusable, modular pipeline components using Azure Data Factorys Data Flows and custom Azure Functions.
- Comprehensive understanding of compliance frameworks (HIPAA) and security using RBAC, Private Link, Key Vault integration, and data masking to secure sensitive data in AI pipelines.
- Configure and maintain data processing platforms like Databricks and Azure Fabric.
Other
- Bachelor's degree or higher in computer science, data science, engineering, mathematics, or a related field with 3 years of experience, (with one year of work experience in an environment where HIPPA compliance is demonstrated.)
- High school diploma or higher with 5 years of equivalent practical work experience, (with 2 years of work experience in an environment where HIPPA compliance is demonstrated.)
- Excellent problem-solving skills and attention to detail
- Strong communication and collaboration skills to work effectively with cross-functional teams
- Experience in a healthcare environment, including familiarity with healthcare data management and regulations
