Lilly is seeking a Senior Data Engineer to implement and optimize large-scale Lakehouse solutions and drive the evolution of their modern data platform, while providing technical leadership to a growing team.
Requirements
- Experience with streaming data technologies (Kafka)
- Familiarity with data cataloging tools (Apache Atlas or DataHub)
- Familiarity with high performance data service framework (Arrow Flight)
- Expert-level proficiency in Python and SQL for data transformation and pipeline development
- Strong experience with Apache Spark for big data processing and analytics
- Hands-on experience with cloud platforms (AWS or Azure) and their data services
- Proficiency with Infrastructure as Code tools (Terraform, CloudFormation)
Responsibilities
- Design and implement comprehensive Lakehouse architecture solutions using technologies like Databricks, Snowflake, or equivalent platforms
- Build and maintain real-time and batch data processing systems using Apache Spark, Kafka, and similar technologies
- Architect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data.
- Develop data transformation workflows using tools like DBT, Airflow, or Databricks
- Lead the technical strategy for data lake and data warehouse integration, ensuring optimal performance and cost efficiency.
- Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols.
- Implement centralized data catalog system and enhance data discovery using technologies like Elastic Search / Open Search.
Other
- Proven ability to mentor junior engineers and facilitate knowledge sharing
- Strong project management skills with experience leading cross-functional initiatives
- Coordinate cross-functional projects and ensure effective communication between technical and business teams
- Demonstrated ability to make architectural decisions and drive technical consensus
- Embrace a growth mindset and actively seek opportunities to expand your leadership capabilities and technical mastery
