Uber's Batch Data team is revolutionizing Uber's data infrastructure, enhancing data freshness and cost efficiency, and developing a cutting-edge metadata management service for unstructured data. The successful candidate will play a pivotal role in designing and implementing solutions that handle exabytes of data, directly impacting critical Uber data platform services, with unified catalog and storage being two primary focus areas. This role offers the opportunity to drive significant improvements in user experience, quality, innovations, cost & operational efficiency, and system reliability across the organization.
Requirements
- Proven experience in designing and building large-scale data storage and unified data catalog solutions.
- Strong understanding of data platform technologies, including data lakehouse architecture and cloud infrastructure (preferably GCP or OCI).
- Demonstrated expertise in data infrastructure components such as data catalog and storage systems.
- Extensive experience with technologies like Apache Spark, Flink, Kafka, and Table Formats (Hudi, Iceberg, Delta).
- Deep knowledge of data catalog/metadata management.
- Experience developing and implementing S3-compatible APIs and managing large-scale tabular and unstructured data.
- Familiarity with multi-cloud environments (OCI/GCP) and strategies for resource efficiency, including developing intelligent scheduling and portable data solutions.
Responsibilities
- Lead the design, development, and deployment of new features for the large scale cloud data storage platform.
- Drive the architecture and implementation of a unified catalog, focusing on creating a lightweight metadata layer, leveraging native cloud object store capabilities, and on behalf of Uber, to engage the community and contribute to open source projects.
- Collaborate on the development and rollout of uFlash, including implementing full S3 API support, integrating with existing clients, and onboarding new customers.
- Contribute to strategic Data for AI initiatives.
Other
- The base salary range for this role is USD$257,000 per year - USD$285,500 per year.
- You will be eligible to participate in Uber's bonus program, and may be offered an equity award & other types of comp.
- You will also be eligible for various benefits.
- Unless formally approved to work fully remotely, Uber expects employees to spend at least half of their work time in their assigned office.
