Qualtrics is looking for a Senior SDET to lead the development of an agentic evaluation framework within the ML team to ensure robust testing processes and tooling for their expanding AI capabilities, standardizing AI agent testing and establishing Qualtrics as a leader in responsible AI development.
Requirements
- 5+ years of professional coding experience with strong Python skills
- 2+ years experience building test automation frameworks and tools from scratch
- 5+ years of relevant experience as part of an automated test team, with 1+ years as a senior contributor
- Understanding of Machine Learning concepts (deep learning, categorization, classification, data mining)
- Experience with Machine Learning evaluation frameworks
- Experience with CI/CD deployment automation (Jenkins, etc.)
- Proficiency with testing distributed systems and SOA (Service Oriented Architecture)
Responsibilities
- Drive the quality assurance strategy for agentic features, ensuring standardization of testing approaches from a customer-centric perspective
- Develop comprehensive evaluation tools and frameworks that enable automated testing of AI agent workflows
- Define model launch readiness criteria for agentic features that align with both customer needs and regulatory requirements
- Establish model risk management protocols that document AI system capabilities and limitations
- Create knowledge transfer processes to disseminate best practices in AI testing across Qualtrics teams
- Design and implement automated evaluation frameworks for testing AI agent behaviors and outputs
- Build upon our existing AI test strategy and LLM-as-a-Judge infrastructure to create comprehensive testing solutions
Other
- Bachelor's degree in Computer Science or related field
- Work in a collaborative environment with ML Engineers, Research Scientists, Software Engineers, and Product Managers
- Expand your technical leadership through cross-functional collaboration and knowledge sharing
- Stay at the forefront of AI testing practices by researching and implementing new techniques
- Collaborate with product teams to integrate quality processes into their AI development workflows
- Present testing results and quality metrics to stakeholders across engineering and product teams
- The Qualtrics Hybrid Work Model: Our hybrid work model is elegantly simple: we all gather in the office three days a week; Mondays and Thursdays, plus one day selected by your organizational leader. These purposeful in-person days in thoughtfully designed offices help us do our best work and harness the power of collaboration and innovation. For the rest of the week, work where you want, owning the integration of work and life.
