Job Board
LogoLogo

Get Jobs Tailored to Your Resume

Filtr uses AI to scan 1000+ jobs and finds postings that perfectly matches your resume

Xai Logo

Software Engineer - Data Acquisition / Web Crawling

Xai

$180,000 - $440,000
Dec 6, 2025
Palo Alto, CA, US • San Francisco, CA, US
Apply Now

xAI is looking to solve the problem of creating AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge by building world-class systems to collect and process hundreds of petabytes of data across diverse modalities.

Requirements

  • Strong proficiency in at least one compiled language: Rust, Go, C++, or Java.
  • Experience with performance optimization of large-scale systems is preferred.
  • Organizing and meticulously bookkeeping data across multiple clouds, of multiple modalities, and from many sources.
  • Experience with SQL/NoSQL databases, especially columnar databases, is a plus.
  • Great debugging skills are a must.
  • Deep knowledge of how the internet works, including DNS, OSI model, crawler architectures, challenges operating crawlers, and headless browsers.
  • Building bespoke data processing libraries from scratch.

Responsibilities

  • Building petabyte-scale, high-throughput data processing systems managing hundreds of petabytes to exabytes of data.
  • Designing and operating large-scale distributed systems and pipelines processing hundreds of thousands to millions of operations per second.
  • Managing workloads across large cloud compute clusters.
  • Pre-processing datasets for AI training.
  • Building and operating large-scale crawlers, gathering and communicating requirements clearly and concisely.

Other

  • Strong communication skills to concisely and accurately share knowledge with teammates.
  • Work ethic and strong prioritization skills are important.
  • Leadership is given to those who show initiative and consistently deliver excellence.
  • Must be located near the Bay Area or open to relocation.
  • Bachelor's, Master's, or Ph.D. degree is not explicitly mentioned but may be required.