HealthVerity is looking to enhance and support its Petabyte-scale core data asset by hiring a senior data engineer to work on the data platform. The goal is to ensure the platform is available, secure, and reliable, while developing scalable engineering solutions that align with the company's product roadmap and vision.
Requirements
- You are proficient in at least one primary language (e.g., Java, Scala, Python) and Advanced SQL (any variant)
- You have experience with Databricks pipeline automation, AWS EMR, AWS S3 service, Snowflake, Spark, Docker
- You have 8+ years of industry experience and proficiency in building distributed data pipelines for both batch and real-time (experience with Spark, Hive, Iceberg, Kafka, Snowflake is helpful, but not strictly required)
- You have a strong knowledge of Databricks features and functionalities, such as Unity Catalog, Audit Logs, Databricks SQL and Delta Live Tables
- Experience with CI/CD pipelines and DataOps
- You are able to successfully lead large, complex systems design and implementation challenges independently
- Experience using Infrastructure as Code (IaC) tools, such as Terraform, YAML, and Helm Charts
Responsibilities
- Engineer efficient, adaptable and scalable data pipelines to process structured and unstructured data
- Develop and maintain data pipelines to efficiently process and analyze large amounts of streaming data
- Collaborate with other data engineers to maintain a cohesive and standardized data infrastructure
- Work closely with the software engineering team to integrate data pipelines into the overall platform architecture
- Identify and implement solutions to optimize data storage, retrieval, and processing
- Continuously evaluate and improve data engineering processes and systems to increase efficiency and scalability
- Leverage large-scale distributed computing and serverless architecture including Spark, AWS Lambda, etc. to develop pipelines for transforming data
Other
- This is a hybrid role requiring 3 days in office at our Philadelphia HQ - 1818 Market Street.
- You have a product mindset to understand business needs and develop scalable engineering solutions
- You are always looking for opportunities to simplify, automate tasks, and build reusable components across multiple use cases and teams
- You have strong communication skills to collaborate with cross-functional partners and drive projects.
- You are curious and eager to work across a variety of engineering specialties (i.e., Data Science, Data Engineering, and Machine Learning to name a few)
