FanDuel is looking to ensure the proper upkeep of its analytic platforms by establishing platform governance and feature improvements to enable the business, and to drive adoption of modern platform features and implement data governance standards across the organization.
Requirements
- 5+ years of development experience in a core programming language, preferably Python, with strong skills in coding, debugging, and testing.
- 5+ years of hands-on experience with Databricks or an equivalent big data platform (e.g., GCP BigQuery, AWS Redshift), including: Building and optimizing ETL/ELT pipelines in Delta Lake or similar data lake formats. Familiarity with Unity Catalog, SQL Warehouses, and performance tuning in Databricks.
- 5+ years of experience using Apache Airflow (preferably with Astronomer or MWAA), including: Designing modular, idempotent, production-ready DAGs. Experience with dynamic DAG generation, task retries, and sensor management.
- Strong experience working in cloud environments like AWS, Azure, or GCP, including: Proficiency in cloud-native services such as S3, IAM, Lambda, EventBridge, and Secrets Manager. Understanding of cloud networking, security, and data storage best practices.
- Experience with Infrastructure as Code tools, preferably Terraform, for managing Airflow and Databricks deployments.
- Familiarity with CI/CD practices, particularly as they relate to data platform tooling (e.g., GitHub Actions, Azure DevOps, Buildkite).
- Exposure to data quality, observability, and lineage tools, such as Monte Carlo or Datafold.
Responsibilities
- Lead regular design and code reviews to enforce development best practices and ensure high-quality, maintainable pipelines across the team.
- Monitor and optimize job and query performance in Databricks and Airflow, identifying bottlenecks and implementing scalable solutions.
- Contribute to the architecture and design of modern data platform solutions leveraging Databricks and Airflow, with a focus on performance, scalability, and cost-efficiency.
- Author and maintain clear documentation on pipeline architecture, operational workflows, data models, and platform configurations.
- Partner with leadership to drive adoption of modern platform features and implement data governance standards across the organization.
- Design and implement production-grade data pipelines for real-time and batch workloads, including infrastructure and operational monitoring.
- Mentor junior engineers by promoting platform best practices, sharing design patterns, and encouraging innovation and ownership.
Other
- Collaboration is at the core of your role.
- You will have established analytic engineer capabilities as well as enablement skills to allow for data process excellence in the organization.
- In this role, you will be working closely with the data users across the organization to ensure the adoption of platform best practices.
- You’ll also be expected to influence the team to invent, evolve, and/or deprecate data solutions.
- You should have a passion for imparting knowledge by working with analysts to improve data processes and to minimize areas of risk across our platforms.
