MCG is looking to improve the US healthcare system by developing evidence-based content and innovating products. The Manager of Data Science will lead a team to effectively test agents, ensuring statistically sound evaluations and accurate reporting to stakeholders.
Requirements
- Expert level Python proficiency
- Ability to write reliable, extensible python packages
- Demonstrated experience translating high level requirements into useful tools
- Excellent knowledge of frequentist and Bayesian statistical techniques
- Experience with basic principles of deep learning and/or natural language processing
- Experience working specifically on automated model evaluation
Responsibilities
- Lead Data Scientists and analysts as they build statistically sound evaluations, guide annotation, and develop ground truths for complex tasks.
- Maintain and expand metrics used to evaluate agent performance and fairness
- Create evaluation packages for Data Scientist and Machine Learning Engineers to use when running experiments
- Ensure accurate, regular reporting on evaluation metrics to stakeholders
- Manage periodic re-evaluation of production agents using user data and annotation
- Direct annotation by managing annotation budgets, sourcing data, and providing statistical analysis of annotations
Other
- Healthcare experience
- Good written communication skills
- Remote work
- Travel expected 3-4 times per year for company-sponsored events
- The ideal candidate should be comfortable balancing the independence of remote work with the collaborative opportunities offered by periodic in-person engagements.
