Grafana Labs is looking for an experienced engineer to help build and evolve internal evaluation frameworks for Generative AI systems, particularly Large Language Models (LLMs), to make sense of complex observability data through AI-driven features.
Requirements
- Experience designing and implementing evaluation frameworks for AI/ML systems.
- Familiarity with prompt engineering, structured output evaluation, and context-window management in LLM systems.
- High autonomy to collaborate and translate team goals into clear, testable criteria supported by effective tooling.
- Experience working in environments with rapid iteration and experimental development.
- A pragmatic mindset that values reproducibility, developer experience, and thoughtful trade-offs when scaling GenAI systems.
- A passion for minimizing human toil and building AI systems that actively support engineers.
Responsibilities
- Design and implement robust evaluation frameworks for GenAI and LLM-based systems, including golden test sets, regression tracking, LLM-as-judge methods, and structured output verification.
- Develop tooling to enable automated, low-friction evaluation of model outputs, prompts, and agent behaviors.
- Define and refine metrics for both structure and semantics, ensuring alignment with realistic use cases and operational constraints.
- Lead the development of dataset management processes and guide teams across Grafana in best practices for GenAI evaluation.
Other
- This is a remote opportunity and we would be interested in applicants from USA time zones only at this time.
- You may not meet every requirement, and that’s okay. If this role excites you, we’d love you to raise your hand for what could be a truly career-defining opportunity.
- In-Person onboarding
- Balance is Key - We operate a global annual leave policy of 30 days per annum. 3 days of your annual leave entitlement are reserved for Grafana Shutdown Days to allow the team to really disconnect.
- We will comply with local legislation where applicable.
