Databricks is looking to scale the foundation of their Vector Search product by designing and evolving distributed systems that serve as the backbone of real-time, AI-powered applications. This involves enabling developers to build a wide spectrum of AI applications like Retrieval-Augmented Generation (RAG) and recommendations, supporting scalable similarity search across diverse datasets, and ensuring exceptional performance, scalability, and reliability for their fastest-growing product.
Requirements
- 15+ years of industry experience building and operating large-scale distributed systems.
- Deep expertise in Search Engines (vector and/or keyword-based), including areas such as indexing, ranking, retrieval infrastructure, and query execution.
- Familiarity with storage systems and database internals.
- Strong foundation in algorithms, data structures, and system design, especially as applied to real-world data processing and retrieval problems.
- Track record of driving high-impact, technically complex initiatives that delivered clear customer or business value.
- Experience leading architecture efforts for performance-sensitive systems (e.g., latency-critical services, multi-tenant platforms, large-scale indexing pipelines).
Responsibilities
- Drive the architecture and evolution of the Vector Search engine, delivering capabilities that support scalable, low-latency search and retrieval across massive, multimodal datasets.
- Identify and solve key challenges in indexing, query execution, storage, and ranking — combining innovations in vector search with traditional information retrieval and database techniques.
- Partner with infrastructure, product, and research teams to shape APIs and developer workflows that make it easy to build and deploy production-grade AI-powered applications.
- Lead critical engineering efforts, from whiteboard design to production rollout, ensuring high standards for performance, correctness, and system resilience.
- Make direct, high-impact contributions to production systems — from designing core components to writing performant, maintainable code that powers search and retrieval at scale.
- Champion software engineering best practices and help raise the technical bar across the organization through mentorship, design reviews, and collaboration.
- Contribute to long-range planning and influence company-wide technical direction in search, AI infrastructure, and data systems.
Other
- Strong communication skills and comfort operating in fast-moving, cross-functional environments.
- A strategic mindset with the ability to align engineering execution with longer-term product and company goals.
- Passion for mentoring and developing other engineers.
