Yahoo needs to design, build, and optimize scalable data pipelines and infrastructure to power advanced analytics and machine learning solutions, leveraging AI/ML experience and distributed systems knowledge to improve user experience and drive business objectives.
Requirements
- 4+ years experience in relevant software development with at least 2 years of professional Java and/or Python experience
- 2+ years experience in the Big Data pipeline and analytics space with experience across technology stacks.
- 2+ years experience in custom ETL design using Big Data stack environments (Hadoop, MapReduce, Pig, Hive, AWS EMR, Apache Beam, Google Cloud Platform Dataflow, BigQuery), implementation and maintenance.
- Experience or familiarity with some of the following tools: Kafka, Storm, Streaming (Spark,Dataflow), ElasticSearch
- Implement and optimize data storage solutions using GCP services such as BigQuery, Cloud Storage, and Dataflow.
- Experience working with either MapReduce or any other Parallel data processing system.
- Experience with schema design and dimensional data modeling.
Responsibilities
- Apply software engineering expertise to build high-performance, scalable data warehouses.
- Design, build, and launch efficient & reliable data pipelines to move and transform data on the scale of multiple petabyte(s) using the latest technologies.
- Build real time analytics and ingestion pipelines capable of processing more than a million events per second and provide insights at sub-second latency
- Design and audit processes for ensuring the delivery of high-quality data through rigorous QA checks
- Design workflows to ingest, load and present new data sets for users.
- Provide active support, be on rotation for on-call support on production pipelines (typically a couple of times each quarter).
- Work with the production engineering / infrastructure team to drive resolution to production issues.
Other
- Interact with product owners and end users to understand and solve new business requirements as they emerge.
- Have excellent data modeling skills to understand the nuances of various dimension and metric types in warehouses.
- Define and manage SLA for all data sets in allocated areas of ownership.
- flexible hybrid work options
- All qualified applicants will receive consideration for employment without regard to, and will not be discriminated against based on age, race, gender, color, religion, national origin, sexual orientation, gender identity, veteran status, disability or any other protected category.
