ServiceNow is looking to improve the quality, accuracy, performance, and safety of its AI/ML products, specifically its Natural Language Understanding and Artificial Intelligence engine, by developing robust testing strategies and frameworks.
Requirements
- Deep, hands-on experience testing machine learning and generative AI systems, including validating model predictions and conducting end-to-end workflow integration testing.
- Strong knowledge of synthetic data generation and prompt engineering to support robust model evaluation.
- Expert-level understanding of AI principles, including agentic AI, algorithms, models, and responsible AI/ethics.
- Proficient in multiple automated testing frameworks such as Java, JUnit, Selenium, TestNG, and other open-source tools.
- Fluent in Java and Python, with strong scripting skills for test automation and process optimization.
- In-depth experience applying a wide range of testing techniques (unit, integration, performance, automation) and choosing the right one for the task.
- Expert user of development tools such as IDEs, debuggers, build tools, source control systems, profilers, and Unix/system admin tools.
Responsibilities
- Work on AI/ML-specific test strategies, and build testing frameworks.
- Take ownership of end-to-end quality deliverables for ML/AI products.
- Evaluate the quality, accuracy, performance, and safety of AI-generated content produced by LLMs and identify weaknesses.
- Create and maintain test automation frameworks.
- Partner with labeling teams to drive data creation processes/validation.
- Collect, analyze, and report quality metrics from test execution and create defects based on results patterns.
- Contribute to refining evaluation criteria and methodologies for model assessment to improve AI evaluation frameworks.
Other
- Act as the voice of quality, collaborating with Product and Development teams to define comprehensive test strategy while establishing and adhering to quality milestones, and execute according to plan.
- Coordinate with engineering organizations in troubleshooting or addressing issues with applications and dev/test environments.
- Stay up to date with latest developments in LLMs and AI ethics to improve evaluation techniques.
- Experience in leveraging or critically thinking about how to integrate AI into work processes, decision-making, or problem-solving.
- Proven ability to lead quality efforts within Agile scrum teams, guiding QE engineers and collaborating cross-functionally.
- Skilled at diagnosing and resolving complex issues, with a focus on root cause analysis (RCA) and long-term solutions.
- Demonstrated ability to implement best practices and coding standards across tools, frameworks, and design patterns.
- Familiar with ServiceNow instances and their integration in quality engineering workflows.
- Solid experience working within Agile development environments, ensuring quality is embedded across the SDLC.
- Effective at leading technical discussions across teams, bringing a clear understanding of software development life cycle (SDLC) best practices.
- 8 years of related experience with a Bachelor's degree; or 6 years and a Master's degree; or a PhD with 3 years experience; or equivalent experience.
- Work personas (flexible, remote, or required in office) are categories that are assigned to ServiceNow employees depending on the nature of their work.
- All qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, national origin or nationality, ancestry, age, disability, gender identity or expression, marital status, veteran status, or any other category protected by law.
- If you require a reasonable accommodation to complete any part of the application process, or are unable to use this online application and need an alternative method to apply, please contact globaltalentss@servicenow.com for assistance.
- For positions requiring access to controlled technology subject to export control regulations, including the U.S. Export Administration Regulations (EAR), ServiceNow may be required to obtain export control approval from government authorities for certain individuals.
- All employment is contingent upon ServiceNow obtaining any export license or other approval that may be required by relevant export control authorities.
