Abbott is looking to solve the problem of designing, building, and maintaining robust data pipelines to help people live healthier lives, with a focus on medical device technologies.
Requirements
- Proficiency in Azure Databricks, PySpark, and SQL is essential.
- Familiarity with Rest API for data extraction and R for data manipulation and analysis is a plus.
- Competence in Python for data manipulation and ETL tasks.
- Experience with the Databricks Lakehouse Platform and Databricks SQL.
- Experience with healthcare data preferred (insurance Claims, EHR and Med Device data, etc).
- Databricks Certified Data Engineer Associate or Professional certification is highly desirable.
- At least 6+ months of hands-on experience performing data engineering tasks with Databricks.
Responsibilities
- Data Pipeline Development: Design, build, and maintain robust data pipelines using Azure Databricks and PySpark, ensuring optimal performance and scalability.
- ETL Implementation: Develop and optimize ETL pipelines utilizing Databricks tools to facilitate efficient data processing and transformation.
- Data Processing: Leverage Apache Spark for both batch and incremental data processing, ensuring data integrity and consistency.
- Collaboration: Work closely with data scientists and analysts to comprehend data requirements and deliver tailored solutions that meet business needs.
- SQL Development: Craft and refine complex SQL queries for effective data extraction, analysis, and reporting.
- R Utilization: Apply R for advanced data manipulation and statistical analysis, enhancing data insights (nice to have).
- Data Governance: Implement stringent data governance practices using tools such as Unity Catalog to ensure data quality, security, and compliance.
Other
- Associates Degree (± 13 yrs) Associates or Bachelor’s degree in a technical field (Science, Technology, Engineering, or Mathematics) OR an equivalent combination of education and work experience or relevant software certification (e.g. Udacity Nanodegree)
- Minimum 4 years of experience in software development (combination of programming-heavy coursework, school projects, freelance work and professional experience)
- Ability to work independently, providing self-guided solutions and demonstrating initiative.
- Exceptional problem-solving skills and meticulous attention to detail.
- Strong communication and collaboration skills, with the ability to work effectively in team settings and engage with stakeholders.
