Material Security is looking to develop high quality data sets that will be used to create ML/AI models that detect security relevant data and behavior (phishing emails, sensitive data in email and drive).
Requirements
- 5+ years of program management experience, ideally in ML ops, data labeling, or AI infrastructure.
- Proven track record building and managing remote labeling teams.
- Strong understanding of ML lifecycle stages and the importance of annotated data quality.
- Experience defining SOPs, audit mechanisms, and workflows for scalable data labeling.
- Proficient in project management tools such as Jira, Asana, or Linear for program tracking
- A deep understanding on ML Operations labelling tools and experience building or maintaining an annotation tool.
- Understanding of data privacy and security standards and how they can be followed in a labeling program.
Responsibilities
- Define and drive end-to-end execution of large-scale annotation programs across multiple data types.
- Collaborate with ML, product, and data operations teams to scope and prioritize labeling needs.
- Own vendor engagement: onboarding, SLA management, training, and quality reviews.
- Build feedback loops between annotators and model performance to inform labeling strategies.
- Create dashboards and reporting mechanisms to track labeling velocity, quality, and cost.
- Lead initiatives to improve labeling efficiency through tooling enhancements and process automation.
- Be the voice of labeling in cross-functional forums-translating model needs into operational plans.
Other
- Manage and mentor a team of trained threat analysts who conduct our labeling.
- Conduct analysis of the quality of the labeling and for insights into how our detections can be improved.
- Hire and train new or replacement threat analysts
- Strong analytical and communication skills; ability to synthesise feedback from ML, ops, and product stakeholders and also analyzed data to spot trends in our labeling or detection quality.
- The ability to develop and maintain labeling quality metrics and analytic insights and report on those to senior management
