Microsoft AI is looking to drive Copilot model capabilities such as tool-use to ensure that the language models that power Microsoft Copilot deliver high quality responses to users whilst being grounded, reliable, and cost-efficient.
Requirements
- Hands-on experience with LLM APIs (e.g. OpenAI, Anthropic, Azure OpenAI), embeddings, vector databases, and tool use.
- Hands-on experience with prompt design, context window management, and model evaluation.
- 2+ years of hands-on experience in building search, recommendation systems, driving measurable improvements in relevance, engagement, and conversion through data-driven experimentation.
- 1+ years of experience in building LLM applications with familiarity in agent and orchestration frameworks, tool use, LLM evaluations and driving efficiency improvements.
- Technical depth in software development, data science and machine learning.
- Experience using and applying Large Language Models.
- Familiarity with LLMs, including how to prompt engineer them, how to evaluate them, and understand how they are tuned.
Responsibilities
- Develop and execute on LLM tools strategy for Copilot that extend language model's capabilities.
- Prototype approaches by steering language models to drive response quality across a wide range of scenarios.
- Identify and prioritize platform, orchestration and language model issues that impact quality, factuality and safety and working with engineers and researchers to find a path to resolution.
- Define and build measurable evaluations with relevant datasets to demonstrate quality improvements.
- Define, deploy and manage experiments in production that impact language model's tool use, driving measurable improvements in relevance for and engagement with Copilot users.
- Partner with product teams to scale tool building and work with inference, agents and orchestration teams to resolve dependencies.
- Accountable to own the status of key projects, proactively identifying risks and proposing solutions to ensure timely delivery.
Other
- Bachelor's Degree AND 5+ years experience in product management OR equivalent experience.
- 3+ years of experience leading ambiguous product areas, defining requirements, developing roadmaps, and working with multi-disciplinary teams to execute them.
- 2+ years of experience building ML-powered or LLM-powered products.
- Ability to work from a designated Microsoft office at least four days a week if they live within 50 miles (U.S.) or 25 miles (non-U.S., country-specific) of that location.
- Positive energy, empathy, and kindness, in addition to being highly effective.
