The company needs to evaluate and improve the performance of AI models in healthcare applications.
Requirements
- Expertise in creating evaluation frameworks for large language models and other AI systems
- Expertise in using and querying relational database management systems such as Microsoft SQL Server, Snowflake or postgreSQL
- Expertise in using BI tools such as Tableau or PowerBI for AI performance visualization
- Experience using Python for AI model evaluation and output analysis
- Demonstrated thought leadership in analytics strategy and enterprise-wide project implementation
- Experience in a data analytics role with focus on AI/ML evaluation
- Experience with AI model evaluation, including performance metrics analysis, output validation, and data visualization
Responsibilities
- Leading requirements gathering and development of analytics frameworks for evaluating generative AI models in healthcare applications
- Defining, developing, and deploying queries and metrics to assess AI model performance and output quality
- Leading data model creation and maintenance for AI evaluation frameworks and monitoring systems
- Conducting comprehensive QA of AI outputs and participating in risk analysis and mitigation
- Understanding how security and privacy laws such as HIPAA affect AI deployment in healthcare
- Understanding how to handle sensitive data including PHI and PII in AI contexts
- Managing the analytics roadmap for model evaluation
Other
- Leading the strategic direction for AI evaluation methodologies in their product area
- Collaborating with internal and external stakeholders to manage evaluation timelines and deliverables
- Collaborating with cross-functional teams including AI researchers, product managers, clinicians, and engineers
- Bachelor's degree required, Master's or PhD preferred in Mathematics/Science
- 6-9 years of relevant experience in data analytics with focus on AI/ML evaluation
