xAI's mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge. The Data Platform team builds and operates the infrastructure responsible for all large-scale data transport and processing across the company, enabling real-time ML pipelines, feed ranking, experimentation, analytics, and observability at petabyte scale.
Requirements
- Proven expertise in distributed systems, stream processing, or large-scale data platforms.
- Proficiency in Rust, Go, Scala or similar systems languages.
- Hands-on experience with Kafka, Flink, Spark, Trino, or Hadoop*in production.
- Strong debugging, profiling, and performance optimization skills.
- Track record of shipping and maintaining critical infrastructure.
- Practical coding assessment in a language of your choice.
- Systems design hands-on: Demonstrate practical skills in a live problem-solving session.
Responsibilities
- Design and implement high-throughput, low-latency data ingestion and transport systems.
- Scale and optimize multi-tenant Kafka infrastructure supporting real-time workloads.
- Extend and tune Spark, Flink, and Trino for demanding production pipelines.
- Build interfaces, APIs, and pipelines enabling teams to query, process, and move data at petabyte scale.
- Debug and optimize distributed systems, with a focus on reliability and performance under load.
- Collaborate with ML, product, and infrastructure teams to unblock critical data workflows.
Other
- All employees are expected to be hands-on and to contribute directly to the company’s mission.
- Leadership is given to those who show initiative and consistently deliver excellence.
- Work ethic and strong prioritization skills are important.
- All engineers and researchers are expected to have strong communication skills.
- Candidates who join in San Francisco must make it to Palo Alto at least twice a week.
