Klaviyo is looking to solve the problem of ingesting, processing, and analyzing billions of data points daily to power core functionality and generate data-driven insights. This involves building and scaling real-time and offline data analytics platforms hosted on AWS.
Requirements
- Hands on with Python and SQL, with experience in backend development.
- Experience with distributed data processing frameworks such as Apache Spark and Flink.
- Proven track record of designing and implementing scalable ETL/ELT pipelines, ideally using AWS services like EMR.
- Strong knowledge of cloud platforms, particularly AWS (e.g., EMR, S3, Redshift), and optimizing data workflows in the cloud.
- Experience with data pipeline orchestration tools like Airflow.
- Familiarity with real-time data streaming technologies such as Kafka or Pulsar.
- Understanding of data modeling, database design, and data governance best practices.
Responsibilities
- Implement scalable, fault-tolerant data pipelines using distributed processing frameworks like Apache Spark and Flink on AWS EMR, optimizing for throughput and latency
- Design batch and real-time, event-driven data workflows to process billions of data points daily, leveraging streaming technologies like Kafka and Flink.
- Optimize distributed compute clusters and storage systems (e.g., S3, HDFS) to handle petabyte-scale datasets efficiently, ensuring resource efficiency and cost-effectiveness.
- Develop robust failure recovery mechanisms, including checkpointing, replication, and automated failover, to ensure high availability in distributed environments
- Optimize data storage and processing systems to handle petabyte-scale datasets efficiently, ensuring performance and cost-effectiveness.
- Collaborate with cross-functional teams to deliver actionable datasets that power analytics and AI capabilities.
- Own the technical direction of highly visible data systems, improving monitoring, failure recovery, and performance.
Other
- 4+ years of experience in software development, with at least 2 years focused on data engineering and distributed systems.
- Excellent problem-solving skills and the ability to thrive in a fast-paced, collaborative environment.
- Strong communication skills with experience mentoring or leading engineering teams.
- You’ve already experimented with AI in work or personal projects, and you’re excited to dive in and learn fast.
- This role is based in Boston, MA and requires a hybrid, in-office component.
