Magnite is looking to hire a Senior Data Engineer to help architect and build systems to process their large volume of data, which underpins their Streaming business. The goal is to ensure these systems remain scalable and efficient while maintaining data consistency and reliability.
Requirements
- Scala for core application development
- Spark (batch + streaming) for data pipeline development, currently based on Databricks
- Terraform, Docker, Jenkins for CI/CD / infra / application deployment
- Airflow for job orchestration
- AWS-based cloud infrastructure including RDS, EC2, S3, Kinesis, ECS
- Designing and building systems that work with large-scale data volumes and data ingestion systems at scale in a cloud-first setting
- Established technical excellence with a desire to learn and crush the data engineering world
Responsibilities
- Help architect and build systems to process our data volume to empower all consumers
- Join in on full end-to-end ownership of the team's data pipeline and related systems for data delivery
- Get to work on handling internet-scale data problems
- Build systems that remain scalable and efficient at this volume of data while also ensuring data consistency and reliability
- Own the applications and infrastructure that make up the Streaming Org's data pipeline, handling ~40 billion events per day at an average of 400-500 thousand per second
- Develop systems that can respond to traffic patterns with peak- and off-peak times and seasonal shifts
- Improve profit margins by lowering the infrastructure cost basis through building efficient systems
Other
- 4+ years of software development experience with a BS/MS Computer Science or equivalent work experience
- Wants to take ownership. We are responsible for the entire software development lifecycle, from requirements gathering to production support
- Engage in critical and creative thinking. We expect engineers to present and discuss tradeoffs to solve the problems we face
- Be a part of and promote our culture of collaboration and mentorship
- Value communication, discussion, and sharing of ideas to come to the best technical solutions to our large-scale data challenges
