LMArena is looking for a Machine Learning Scientist to advance how the world evaluates and understands AI models in the open by designing, running, and sharing new methods and experiments that reveal what makes models useful, trustworthy, and capable.
Requirements
- Hands-on experience training large-scale models, including reward models, preference models, and fine-tuning LLMs with methods like RLHF, DPO, and contrastive learning.
- Strong foundation in ML and statistics, with a track record of designing novel training objectives, evaluation schemes, or statistical frameworks to improve model reliability and alignment.
- Fluent in the full experimental stack, from dataset design and large-batch training to rigorous evaluation and ablation, with an eye for what scales to production.
- Strong understanding of LLMs and modern deep learning architectures (e.g., Transformers, diffusion models, reinforcement learning with human feedback)
- Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow
- Demonstrated ability to design and analyze experiments with statistical rigor
- Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation
Responsibilities
- Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions
- Develop new metrics, methodologies, and evaluation protocols that go beyond traditional benchmarks
- Analyze large-scale human voting and interaction data to uncover insights into model performance and user preferences
- Communicate results with the broader research community via academic papers, educational content, conference talks
- Collaborate with engineers to implement and scale research findings into production systems
- Prototype and test research ideas rapidly, balancing rigor with iteration speed
- Partner with model providers to shape evaluation questions and support responsible model testing
Other
- Deeply collaborative mindset, working closely with engineers to productionize research insights and iterating with product teams to align research with user needs.
- Comfortable being a visible representative of LMArena, engaging openly with the research community, and building a strong personal brand to help shape AI research culture.
- Uses personal and professional platforms to amplify open research initiatives and invite collaboration.
- Ability to translate research questions into practical systems and collaborate across engineering and product teams
- Passion for open science, reproducibility, and community-driven research
