At Lilly, we unite caring with discovery to make life better for people around the world. We are seeking a Senior Data Engineer to implement and optimize large-scale Lakehouse solutions and drive the evolution of our modern data platform while providing technical leadership to a growing team.
Requirements
- Experience with streaming data technologies (Kafka*)
- Familiarity with data cataloging tools (Apache Atlas or DataHub*)
- Familiarity with high performance data service framework (Arrow Flight*)
- Expert-level proficiency in Python and SQL* for data transformation and pipeline development
- Strong experience with Apache Spark* for big data processing and analytics
- Hands-on experience with cloud platforms (AWS* or Azure*) and their data services
- Proficiency with Infrastructure as Code tools (Terraform*, CloudFormation*)
Responsibilities
- Design and implement comprehensive Lakehouse architecture solutions using technologies like Databricks, Snowflake, or equivalent platforms
- Build and maintain real-time and batch data processing systems using Apache Spark, Kafka, and similar technologies
- Architect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data.
- Develop data transformation workflows using tools like DBT, Airflow, or Databricks
- Lead the technical strategy for data lake and data warehouse integration, ensuring optimal performance and cost efficiency.
- Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols.
- Implement centralized data catalog system and enhance data discovery using technologies like Elastic Search / Open Search.
Other
- Proven ability to mentor junior engineers and facilitate knowledge sharing
- Strong project management skills with experience leading cross-functional initiatives
- Coordinate cross-functional projects and ensure effective communication between technical and business teams
- Demonstrated ability to make architectural decisions and drive technical consensus
- Embrace a growth mindset and actively seek opportunities to expand your leadership capabilities and technical mastery
