OnCorps LLC is looking to revolutionize the financial services sector by enhancing efficiency, security, and customer satisfaction through advanced AI and machine learning technologies. The ML QA Analyst role is crucial for ensuring the continuous learning, improvement, and performance of their AI systems.
Requirements
- Familiarity with LLMs/Agents is a plus
- Knowledge of SQL is a plus
- Knowledge of Python is a plus
Responsibilities
- Owning data labeling for PDF / Excel-based ML ingestion engines across the company. This includes both training-based labeling and live-run corrective labeling.
- Review evaluation jobs where agents are underperforming, digging into individual logs and traces and providing documentation of possible prompt weaknesses.
- Interacting with customer facing “chat” agents. Validating the integrity of an agent’s response to both “happy-path” and adversarial prompts pre-deployment & when model updates are in testing.
- Owning client-specific dashboard deployments. Working with product/CS stakeholders to deploy client-specific visualizations. Testing dashboards for errors / logical gaps.
- Initiating and monitoring routine model retraining for models in a steady-state & fixing job failures in production. Working with Data Science to determine appropriate thresholds for escalation.
- working to add comments to existing AI Table Annotation.
Other
- STEM BA/BS
- High attention to detail
- Ability to think critically and analytically
- Ability to follow and understand a complex ruleset
- Comfortable working in fast-paced environments and meeting tight deadlines
- U.S. based with work authorization
- Primarily desk-based role during standard hours, utilizing PC, video, and phone
- Requires flexibility to work in-office and remotely based on business needs
