The company is seeking to evaluate and improve the performance of generative AI models in healthcare applications.
Requirements
- Expertise in creating evaluation frameworks for large language models and other AI systems
- Expertise in using and querying relational database management systems such as Microsoft SQL Server, Snowflake or postgreSQL
- Expertise in using BI tools such as Tableau or PowerBI for AI performance visualization
- Experience using Python for AI model evaluation and output analysis
- Experience with AI model evaluation, including performance metrics analysis, output validation, and data visualization
- Experience with data analytics and focus on AI/ML evaluation
- Experience with data model creation and maintenance for AI evaluation frameworks and monitoring systems
Responsibilities
- Leading requirements gathering and development of analytics frameworks for evaluating generative AI models in healthcare applications
- Defining, developing, and deploying queries and metrics to assess AI model performance and output quality
- Leading data model creation and maintenance for AI evaluation frameworks and monitoring systems
- Conducting comprehensive QA of AI outputs and participating in risk analysis and mitigation
- Managing the analytics roadmap for model evaluation
- Collaborating with internal and external stakeholders to manage evaluation timelines and deliverables
- Understanding how security and privacy laws such as HIPAA affect AI deployment in healthcare
Other
- 6-9 years of relevant experience in data analytics with focus on AI/ML evaluation
- Bachelor's degree required, Master's or PhD preferred in Mathematics/Science
- Demonstrated thought leadership in analytics strategy and enterprise-wide project implementation
- Health Benefits: Comprehensive, multi-carrier program for medical, dental and vision benefits
- Retirement Benefits: 401(k) with match and an Employee Share Purchase Plan
- Wellbeing: Wellness platform with incentives, Headspace app subscription, Employee Assistance and Time-off Programs
