Jump is looking for a QA engineer to improve their AI/ML systems by coordinating and running data labeling/annotation campaigns, evaluating production system outputs, and ensuring customers receive accurate transcripts, summaries, and action items.
Requirements
- Curiosity and aptitude to learn ML/AI evaluation (prompt testing, golden sets, offline evals, safety/guardrails)
- Familiarity with AI prompts, LLMs, and the Jump product (as a user or employee)
- Comfortable reading software system logs and finding patterns in messy data
- Experience with BigQuery or other data warehouses
- Experience with web API testing
- Basic familiarity with query languages, relational databases, and other data storage systems
- Designed and executed AI evaluation workflows (golden datasets, human-in-the-loop scoring, clear rubrics)
Responsibilities
- Serve as the embedded QA engineer on two pods (Jump’s cross-functional teams), collaborating with product managers to evaluate AI outputs, run exploratory and regression testing, and unblock engineers and PMs.
- Learn and track AI/ML quality signals, including golden datasets, prompt/regression suites, and metrics such as WER, diarization accuracy, action-item precision/recall, summary faithfulness, hallucination rate, and PII handling.
- Build dashboards for quality KPIs (defect escape rate, flake rate, regression coverage, MTTD/MTTR, AI eval scores) and drive continuous improvement.
- Partner with Product and Engineering to ensure requirements are testable, edge cases are captured, and AI evaluation rubrics are clear and repeatable.
- Foster a no-drama, direct-and-kind culture that moves with high-quality velocity.
- Designed and executed AI evaluation workflows (golden datasets, human-in-the-loop scoring, clear rubrics)—a plus but not required; candidates with this experience may be considered for higher compensation
- Created risk-based test plans and lightweight automation that caught regressions early
Other
- 3+ years in QA or Quality Engineering for SaaS products
- Strong exploratory testing skills and clear, concise written communication for reproducing issues
- You don’t need a traditional STEM background to excel here. You’ll thrive if you Get excited about spotting patterns
- Have a strong grasp of human language and thought processes
- You might have a background in Editing, Technical writing
