Spring Health is looking to revolutionize mental healthcare by removing barriers to access. The company aims to ensure its AI systems are safe, reliable, and effective, requiring a Data Scientist II on the AI Trust team to conduct critical analyses and experiments to measure the real-world impact of AI.
Requirements
- Proficiency in Python and a solid understanding of core statistical concepts. You have a proven ability to write and review production-quality code
- Proven experience in evaluating machine learning models, with exposure to large language models (LLMs) being a strong plus.
- Analyzing A/B tests or other experiments with statistical rigor.
- Using evaluation tools (e.g., LangSmith, open-source libraries) to iteratively measure and improve model performance.
- Building data pipelines or tools to enable collaboration on test sets.
- Applied knowledge of concepts in AI ethics, such as fairness, bias, and interpretability.
Responsibilities
- Own and evolve the evaluation frameworks for our AI and ML models, translating high-level trust principles into specific, measurable tests.
- Define and conduct rigorous experiments to resolve ambiguous questions about the safety, reliability, and impact of our models.
- Collaborate with engineering partners to design and build production-quality code, creating automated, scalable, and pragmatic testing frameworks based on modern best practices.
- Partner with product, legal, and infrastructure teams to implement and monitor standards for trustworthy AI.
- Proactively identify gaps and develop novel evaluation approaches, which may include creating synthetic test data from user traces or building lightweight processes for non-technical partners to iterate on test sets.
- Synthesize complex evaluation results and industry trends into actionable insights and clearly communicate findings to diverse technical and non-technical stakeholders.
Other
- Candidates for this position must be based in the Salt Lake City metro area and be willing to commute 2-3 days a week when this role transitions to a hybrid schedule in 2026.
- Exceptional communication and teamwork skills, with a proven ability to collaborate effectively with diverse, cross-functional teams.
- A strong interest in applying data science to complex, high-stakes domains like mental healthcare. You are motivated by our mission to remove every barrier to mental health.
- A pragmatic and proactive approach to problem-solving, with a history of developing creative solutions to complex problems.
- An avid learning mindset and a passion for staying at the forefront of trends in AI safety, evaluation, and reliability.
