Roblox is looking to solve the problem of evaluating and improving the performance of GenAI systems, ensuring safety, quality, and iteration speed, and enabling Roblox Creators to accelerate their workflows and bring GenAI capabilities to millions of users.
Requirements
- Familiarity with GenAI models and GenAI evaluation methods.
- Proficiency in SQL, Hive, or Spark for transforming and manipulating large datasets.
- Experience with scripting languages such as Python or R.
- Ability to effectively use AI tools to enhance productivity in research, ideation, coding, and documentation.
- Strong learning agility, experience conducting applied research or writing technical papers is a plus.
- A demonstrated track record of solving open-ended data science and modeling problems that drive business impact and improve user experience.
- An advanced Degree and/or PhD in Statistics, Economics, Operations Research, Computer Science, Applied Math, Physics, Engineering, or another quantitative field.
Responsibilities
- Develop and improve evaluation frameworks for GenAI features (text, image, 3D, 4D, agentic workflow), including eval experiment design, eval dataset design, label reliability analysis, results analysis, and online evaluation based on user behavior and feedback.
- Establish best practices and guidelines for GenAI evaluation.
- Conduct product analytics, online experiments (A/B tests) and causal analyses to quantify GenAI feature impact and identify opportunities.
- Build automated evaluation systems, such as research and implement LLM-as-judge and VLM-as-judge methods.
- Research and apply state-of-the-art methodologies in GenAI evaluation.
- Advance reproducible evaluation tooling to lift evaluation rigor and efficiency at the company.
- Proactively explore and develop solutions to improve the AI model and data generation flow, ensuring high-quality input for training and deployment.
Other
- 5+ years of experience in data science or a related field.
- Passion for the GenAI field and enthusiasm for continuously improving methods and practices to drive product quality and business impact.
- Ability to work in a dynamic, collaborative environment.
- Self-starter who is curious, rigorous, and passionate about building innovative solutions that deliver real business value.
- Optional presence on Monday and Friday, with onsite presence required on Tuesday, Wednesday, and Thursday for roles based at headquarters in San Mateo, CA
