Today’s data platforms are built on top of tools made for spreadsheet-like analytics, not the petabytes of multimodal data that power AI. As a result, teams waste months on brittle infrastructure instead of conducting research and building their core product.
Requirements
- strong foundation in systems programming and ideally experience with building distributed data systems or databases (e.g. Hadoop, Spark, Dask, Ray, BigQuery, PostgreSQL etc)
- 3+ years of experience working with distributed data systems (query planning, optimizations, workload pipelining, scheduling, networking, fault tolerance etc)
- Strong fundamentals in systems programming (e.g. C++, Rust, C) and Linux
- Familiarity and experience with cloud technologies (e.g. AWS S3 etc)
Responsibilities
- Planning/Query Optimizer: intelligently optimize users’ workloads with modern database techniques
- Execution Engine: improve memory stability through the use of streaming computation and more efficient data structures
- Distributed Scheduler: improve Daft’s resource utilization, task scheduling and fault tolerance
- Storage: improve Daft integrations with modern data lake technologies such as Apache Parquet, Apache Iceberg and Delta Lake
- Our goal is to build the world’s best open-source distributed query engine, becoming the leading framework for data engineering and analytics.
- We are a young startup - so be prepared to wear many hats such as tinkering with infrastructure, talking to customers and participating heavily in the core design process of our product!
Other
- Please note we're looking for individuals who are excited to be a part of a tight-knit team working together 4 days / week in our SF Mission district office.
- we value engineers who can autonomously scope and solve difficult technical challenges.
- Most importantly, we are looking for someone who works well in small, focused teams with fast iterations and lots of autonomy.
- If you are passionate, intellectually curious and excited to build the next generation of distributed data technologies, we want you on the team!
