eBay's AI Platform team is building the next generation of agentic and inference technologies that power AI experiences for hundreds of millions of users worldwide. They are seeking an ML Inference Router Engineer to design and build a highly scalable, low-latency inference gateway capable of supporting billions of daily requests.
Requirements
- 10+ years of experience building large-scale, fault-tolerant, high-performance distributed systems.
- Strong programming skills in one or more of Java, Go, Rust, or C++ (Java preferred for gateway services).
- Deep understanding of networking, concurrency, memory management, and performance tuning in production systems.
- Proven experience designing and operating low-latency APIs at very large scale (10M+ QPS).
- Hands-on experience with Kubernetes, service meshes, and container orchestration at scale.
- Strong background in cloud infrastructure (AWS, GCP, Azure) and distributed system design.
- Experience with inference serving frameworks (vLLM, Triton, TensorRT-LLM, FasterTransformer, DeepSpeed-MII, or similar).
Responsibilities
- Design and build an LLM inference gateway that scales to billions of daily requests with millisecond-level latency.
- Develop intelligent request routing, load balancing, and fallback mechanisms across heterogeneous LLM backends (internal and external).
- Optimize throughput, cost, and reliability of inference workloads in multi-tenant environments.
- Collaborate with platform, research, and product teams to integrate new models and agentic capabilities into the gateway.
- Implement observability, tracing, and autoscaling for inference traffic across Kubernetes-based clusters.
- Conduct design and code reviews to ensure high standards in distributed systems architecture.
- Stay current with advances in LLM serving, inference acceleration, and model APIs to continuously evolve the platform.
Other
- LI-Hybrid
- The total compensation package for this position may also include other elements, including a target bonus and restricted stock units (as applicable) in addition to a full range of medical, financial, and/or other benefits (including 401(k) eligibility and various paid time off benefits, such as PTO and parental leave).
- If hired, employees will be in an “at-will position” and the Company reserves the right to modify base salary (as well as any other discretionary payment or compensation program) at any time, including for reasons related to individual performance, Company or individual department/team performance, and market factors.
- eBay is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, sex, sexual orientation, gender identity, veteran status, and disability, or other legally protected status.
- If you have a need that requires accommodation, please contact us at talent@ebay.com.
