Apple's Video Engineering Data Analytics and Quality group is seeking an expert to evaluate machine learning and deep learning models, including foundation models and multimodal systems, to deliver amazing user experiences
Requirements
- Advanced proficiency in at least one scripting language, preferably Python
- Hands-on experience evaluating LLMs (e.g., GPT, Claude, PaLM) and using them as scoring/judging mechanisms
- Familiarity with multimodal models (e.g., image + text, video + audio) and related evaluation challenges
- Proficiency in Python and libraries such as NumPy, pandas, scikit-learn, PyTorch, or TensorFlow
- Solid understanding of statistical testing, sampling, confidence intervals, and metrics (e.g., precision/recall, BLEU, ROUGE, FID, etc.)
- Experience working with open-source evaluation tools like OpenEval, ELO-based ranking, or LLM-as-a-Judge frameworks
- Familiarity with prompt engineering, few-shot or zero-shot evaluation techniques
Responsibilities
- Develop robust methodologies to assess the performance of foundation models (e.g., LLMs, vision-language models, etc.) across diverse tasks
- Leverage LLMs as judges to perform subjective and open-ended model evaluations (e.g., for summarization, reasoning, or multimodal generation tasks)
- Build, curate, and lead evaluation datasets and benchmarks
- Conduct failure analysis and uncover edge cases to improve model robustness
- Contribute to our tools and infrastructure to automate and scale evaluation processes
- Collaborate with research, engineering, and product teams to define evaluation goals aligned with user experience and product quality
- Build, curate, and lead evaluation datasets and benchmarks
Other
- BS and a minimum of 10 years relevant industry experience
- Strong documentation skills, including the ability to write technical reports and present to non-technical audiences
- Strong interpersonal skills
- Ability to collaborate with teams at Apple passionate about developing foundation models
- Ability to work with diverse teams, including ML engineers, data scientists, and ML Infrastructure engineers
