The Workflow Orchestration team at Airbnb is responsible for the data workflow orchestration solution, which supports a wide range of data use cases from offline data analytics, experiments, to AI/ML model developments and trainings. The team is looking for an individual to architect, design, and implement scalable, reliable, and highly available data infrastructure solutions to support this critical function.
Requirements
- 5+ years of experience in software engineering, with a focus on data infrastructure or supporting data engineers and/or data scientists.
- Strong programming skills in one or more of the following languages: Java and Python.
- Experience building data applications that create and implement higher-level abstractions on top of the lower-level data infrastructure to simplify complex data operations, improve scalability, and enhance overall system performance.
- Experience with data modeling, database design and various SQL dialects.
- Experience with containerization technologies such as Docker and Kubernetes.
- Extensive background in workflow orchestration solutions such as Apache Airflow, Luigi, Azkaban, Oozie, Prefect or Kubeflow.
- Utilize experience in Data Engineering to optimize processes and bring forth new ideas and technologies that enhance our data capabilities.
Responsibilities
- Architect, design, and implement scalable, reliable, and highly available data infrastructure solutions to support workflow orchestration across the company.
- Develop and maintain automation tools, streamlining deployment and management of the workflow orchestration platform.
- Collaborate with infrastructure engineers, data engineers, and other users of the data warehouse to ensure efficient integration and smooth operation of our platform with other systems and services.
- Conduct performance testing and help optimize our orchestration platform to meet the evolving needs of internal users.
- Participate in technical discussions, code reviews, and pair programming sessions to foster knowledge-sharing and team growth.
- Provide input on best practices to improve scalability, reliability, and usability for the data platform.
Other
- Manage and prioritize multiple projects and tasks, communicating progress and status with key stakeholders.
- Excellent communication skills and the ability to collaborate with cross-functional teams to build empathetic tools and platforms.
- Excellent analytical and problem solving skills.
- BS/MS/PhD in Computer Science, a related field, or equivalent work experience preferred.
- This position is US - Remote Eligible.
