LMArena is looking for a Machine Learning Scientist to lead open-source research, including open data set and code releases, advancing how the world evaluates and understands AI models in the open. The goal is to design, run, and share new methods and experiments that reveal what makes models useful, trustworthy, and capable, grounded in human preference signals and released openly for the full ecosystem and research community to build upon.
Requirements
- Fluent in the full experimental stack, from dataset design and large-batch training to rigorous evaluation and ablation, with an eye for what scales to production.
- Strong understanding of LLMs and modern deep learning architectures (e.g., Transformers, diffusion models, reinforcement learning with human feedback)
- Proficiency in Python and ML research libraries such as PyTorch, JAX, or TensorFlow
- Demonstrated ability to design and analyze experiments with statistical rigor
- Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation
- Comfortable working with real-world usage data and designing metrics beyond standard benchmarks
- Ability to translate research questions into practical systems and collaborate across engineering and product teams
Responsibilities
- Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions
- Develop new metrics, methodologies, and evaluation protocols that go beyond traditional benchmarks
- Analyze large-scale human voting and interaction data to uncover insights into model performance and user preferences
- Communicate results with the broader research community via academic papers, educational content, conference talks
- Collaborate with engineers to implement and scale research findings into production systems
- Prototype and test research ideas rapidly, balancing rigor with iteration speed
- Partner with model providers to shape evaluation questions and support responsible model testing
Other
- PhD or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field
- Uses personal and professional platforms to amplify open research initiatives and invite collaboration.
- Deeply collaborative mindset, working closely with engineers to productionize research insights and iterating with product teams to align research with user needs.
- Comfortable being a visible representative of LMArena, engaging openly with the research community, and building a strong personal brand to help shape AI research culture.
- Passion for open science, reproducibility, and community-driven research
