Evolent Health is seeking an ML/LLM Operations Engineer to ensure their AI systems deliver consistent, reliable, and compliant results in healthcare settings, addressing the challenges of a fragmented healthcare system.
Requirements
- 3+ years of experience in MLOps, DevOps, or similar operational roles supporting AI/ML systems
- Strong proficiency in Python and experience with ML/AI frameworks (PyTorch, TensorFlow, Hugging Face, etc.)
- Experience with monitoring tools and practices for AI systems, including performance metrics, drift detection, and alerting
- Knowledge of containerization technologies like Docker and Kubernetes for deploying and scaling AI services
- Experience with cloud environments (AWS, Azure) for AI deployments
- Knowledge of CI/CD pipelines and automation for AI model deployment
- Experience with logging and monitoring tools (Prometheus, Grafana, ELK stack, etc.)
Responsibilities
- Build and maintain comprehensive monitoring systems for deployed AI/ML models to track performance, detect drift, and alert the team to anomalies
- Develop standardized evaluation frameworks to consistently measure AI feature performance across relevant healthcare metrics
- Oversee regulatory compliance processes, including documentation for bias assessments, model cards, and audit trails required in healthcare
- Support the transition from successful POCs to production-ready services with testing, validation, and monitoring infrastructure
- Configure and maintain Docker container environments for AI microservices
- Create and maintain documentation, runbooks, and operational procedures for all deployed AI systems
- Coordinate with DevOps on infrastructure requirements and optimization
Other
- Bachelor's or master's degree in computer science, data science, or related field
- Familiarity with healthcare compliance requirements for AI systems (preferred)
- Understanding of LLM evaluation metrics and techniques
- Experience with API development and maintenance (FastAPI, Flask, etc.)
- Excellent documentation skills and attention to detail
