Arbitration Forums is looking to safeguard the integrity of AI systems by ensuring that only reliable AI solutions are deployed, and to drive business value through fully tested AI solutions.
Requirements
- Advanced programming knowledge, including mastery of programming languages such as Python and Java.
- Deep understanding of machine learning and AI principles, from generative models to predictive advanced algorithms.
- Cloud computing and knowledge for deploying and managing AI applications on cloud platforms like Microsoft Azure.
- Data management knowledge, including data pre-processing, augmentation, and generation of synthetic data, including the cleaning, labeling, and augmenting of data to train and improve AI models.
- Proficiency in programming languages such as Python, R, and familiarity with AI frameworks (e.g., TensorFlow, Selenium, PyTorch).
- Working knowledge of testing frameworks, from techniques to automate web applications to validation of machine learning models, including Model Context Protocols and SHAP.
- Experience with continuous integration and delivery in the context of DevOps and MLOps.
Responsibilities
- Design, develop, and implement comprehensive test plans to ensure AI system functionality aligns with specifications.
- Conduct automated and manual tests to evaluate the AI system’s performance under different scenarios and deployment methods.
- Collaborate with Data Scientist and GenAI Engineers to pinpoint issues and refine the AI system.
- Document test results and feedback into the development cycle for continuous improvement.
- Stay abreast of new testing tools and methodologies to enhance testing efficacy and cost effectiveness.
- Ensuring the AI system adheres to regulatory standards and ethical considerations.
- Partner with the AI Engineers, MLOps Engineer, and other stakeholders to establish and implement observability and monitoring frameworks to adequately and timely identify degradations and potential ethical/bias issues.
Other
- Bachelor’s or Master’s degree in Computer Science, Information Systems, Data Science, Linguistics, or a related field.
- Minimum of 7 years of experience in AI solution testing, software quality assurance, data governance, data science, or a related role.
- Excellent analytical and problem-solving abilities.
- Strong communication and interpersonal skills to collaborate with cross-functional teams.
- Ability to lead projects and mentor junior staff.
