Yahoo is seeking to build the next generation of their cloud-first data ecosystem and is looking for a Senior Data Engineer to help migrate existing pipelines and design new data solutions from the ground up.
Requirements
- Cloud Ecosystems: Deep expertise in GCP and/or AWS data services
- Data Warehouses / Lakehouses: Experience with modern cloud warehouses like BigQuery, Snowflake, Redshift, Databricks, or Dremio.
- Storage: Proficiency with Google Cloud Storage (GCS) and Amazon S3, Relational, Document and Wide Column Databases, columnar formats (parquet, orc, etc), schemas (avro, protocol buffers, etc), compression formats (gzip, snappy, bzip, etc)
- Processing: Hands-on experience with distributed data processing engines such as Apache Flink, Apache Beam, Spark, or similar technologies
- Runtimes & Orchestration: Expertise in technologies like Cloud Functions, Cloud Run, Kubernetes, Dataflow, Glue, EMR, Dataproc, and orchestration tools like dbt or BigQuery Dataforms and Airflow / Cloud Workflows / Step Functions
- Streaming: exposure and practical experience with streaming platforms like Pub/Sub and Kafka
- Python or Java
Responsibilities
- Design and develop robust, scalable, and resilient data pipelines using both streaming and batch processing technologies.
- Lead the migration of legacy data pipelines to a modern, cloud-first architecture on GCP and AWS.
- Collaborate with analytics and reporting teams to understand their needs and deliver well-structured data models.
- Architect and implement a variety of data solutions, by applying concepts such as Lambda, Kappa, and streaming architectures, ETL/ELT.
- Propose, prototype, and implement new technologies and best practices to improve our data ecosystem's efficiency and reliability.
- Contribute to the full data lifecycle, from data ingestion and transformation to modeling and warehousing.
- Help define and enforce best practices for data quality, governance, and architecture.
Other
- A strong background in data engineering, with a proven track record of designing and building scalable data platforms.
- A proactive self-starter who can take the lead on projects, sell new ideas, and drive them to completion with a high degree of autonomy.
- Excellent communication skills, capable of working with both technical and non-technical stakeholders.
- A fundamental belief in leveraging AI and machine learning as a core component of every data solution, from design to implementation.
- Bachelor's degree or higher in Computer Science, Engineering, or related field (not explicitly mentioned but implied)
