Reddit is looking to hire a Senior Data Engineer to lead the development of data pipelines and workflows for large-scale ML models, impacting various teams like Growth, Ads, Feeds, and Core Machine Learning.
Requirements
- Experience with large scale data transforms to prepare graph data
- Experience with Graph DB, Spark, Kafka pipelines
- Experience working with Airflow and MLFlow
- Experience with storage frameworks like BQ, parquet, iceberg
- Awareness of ML models and architectures is a huge plus.
- Strong focus on scalability, reliability, performance, and ease of use.
- You are an undying advocate for platform users and have a deep intuition for the machine learning development lifecycle.
Responsibilities
- Design and implement scalable and secure data processing pipelines and storage environments that prepare our source of truth datasets for our models.
- Ensure data is cleansed, mapped, transformed, and otherwise optimized for storage and use according to business and technical requirements.
- Build effective data pipelines and workflows to streamline data ingestion, processing, and distribution tasks.
- Setting up and operating data workflow management tools for SQL code versioning, dependency tracing, etc
- Load transformed data into storage and reporting structures in destinations including data warehouse, reporting systems and analytics applications.
- Monitor and troubleshoot issues with the data environment to maintain high availability and performance.
- Support monitoring and observability across training datasets, model metrics and implement diagnostic tools for metric movements.
Other
- 5+ years of experience in Data Engineering or ML Infrastructure
- Strong organizational & communication skills
- This job posting may span more than one career level.
- Reddit is proud to be an equal opportunity employer, and is committed to building a workforce representative of the diverse communities we serve.
- Reddit is committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures.
