Disney Entertainment & ESPN Product & Technology (DEEP&T) is looking to solve complex and distinctive technical problems by developing and implementing groundbreaking products and techniques that shape industry norms. The Big Data Infrastructure team specifically aims to build and maintain large-scale data systems with varied requirements and large storage capabilities, focusing on virtualization, cost efficiency, resiliency, and performance of big data services.
Requirements
- Experience in building in-house big data infrastructure.
- Experience in developing and optimizing big data components (e.g. Spark, Flink, Presto/Trino, etc)
- Experience with modern data formats (Iceberg, Delta, Hudi etc)
- Experience with CICD, fine-tuned metrics, security and compliance enhancement
- Experience in building in-house big data infrastructure.
- Experience in contributing big data components (e.g. HDFS, Hive, Spark, Flink, Presto/Trino, etc)
- Experience in container tech stack, including Kubernetes, Docker, Volcano, etc
Responsibilities
- Develop, scale, and improve in-house/cloud and open-source big data engines (e.g. Spark, Flink, Presto/Trino, etc).
- Investigate new big data technology, and apply it to the DisneyStreaming production environment.
- Build next-gen cloud-based big data infrastructure for batch and streaming data applications, and continuously improve performance, scalability and availability
- Handle architectural and design considerations such as performance, scalability, reusability, and flexibility issues.
- Advocate engineering best practices, including the use of design patterns, code review, and automated unit/functional testing.
- Work together with other engineering teams to influence them on big data system design and optimization.
- Define and lead the adoption of best practices and processes. Collaborate with senior internal team members and external stakeholders to gather requirements and drive implementation
Other
- Ability to drive a project from end to end, including clarifying requirements, resolving conflicts, handling tech challenges and delivering the results
- Motivation to dive deep and become an expert in one or more big data areas. Ready to unpack and crack open-source software to fix bugs or develop new features, and contribute back to the community
- 7+ years of relevant professional experience and Bachelor’s degree in Computer Science or related field OR 5+ years of relevant professional experiences and a Master’s degree in Computer Science or related field
- Experience in managing a big data cluster with over 1000 nodes.
- Collaborate efficiently with Product Managers and other developers to build datastores as a service.
