Help evaluate and improve the quality of tools and products that apply AI to improve access to accurate and helpful information at Google, specifically for Gemini's factuality and quality on information-seeking prompts
Requirements
- Master's degree in Statistics, Data Science, Mathematics, Physics, Economics, Operations Research, Engineering, a related quantitative field, or equivalent practical experience
- 5 years of work experience using analytics to solve product or business problems, coding (e.g., Python, R, SQL), querying databases or statistical analysis
- Experience with LLM evaluation, including autoraters and log analysis
- Coding skills (e.g., Python, R, SQL)
- Querying databases or statistical analysis skills
- Experience with data infrastructure or existing data models
- Familiarity with mathematical models and evaluation metrics
Responsibilities
- Collaborate with stakeholders in cross-projects and team settings to identify and clarify business or product questions to answer, and provide feedback to translate and refine business questions into tractable analysis, evaluation metrics, or mathematical models
- Use custom data infrastructure or existing data models as appropriate, and design and evaluate models to mathematically express and solve defined problems with limited precedent
- Gather information, business goals, priorities, and organizational context around the questions to answer, as well as the existing and upcoming data infrastructure
- Own the process of gathering, extracting, and compiling data across sources via relevant tools (e.g., SQL, R, Python)
- Format, re-structure, or validate data to ensure quality, and review the dataset to ensure it is ready for analysis
- Ensure that models are as accurate as possible without compromising any other response quality aspect, and that response structure and organization are helpful
- Define factuality metrics, eval templates, and build high-quality autoraters for factuality evaluations in Gemini
Other
- Master's degree or PhD degree in a related field
- 5 years of work experience or 3 years of work experience with a PhD degree
- 8 years of work experience or 6 years of work experience with a PhD degree (preferred)
- Ability to work in a team setting and collaborate with stakeholders
- Commitment to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status
