The Product Analyst, AI Solutions is responsible for shaping, optimizing, and maintaining advanced AI-driven features and user experiences within our product suite, ensuring AI systems are accurate, trustworthy, and aligned with client needs, while providing maximum business value.
Requirements
- Experience with AI/ML applications (LLMs, NLP, machine learning pipelines).
- Strong understanding of prompt engineering and generative AI interaction design.
- Basic knowledge of data modeling, relational database concepts, and SQL.
- Familiarity with model evaluation techniques and performance metrics.
- Proficiency with Python or JavaScript for prototyping and evaluation.
- Familiarity with RAG (retrieval-augmented generation), embeddings, MLOps, and responsible AI practices.
- Experience with fine-tuning methods (LoRA, QLoRA, PEFT), running structured evaluation series, or human-in-the-loop systems.
Responsibilities
- Design, evaluate, and refine prompts, templates, and conversation flows for reliability, cost effectiveness, and user alignment.
- Maintain prompt libraries, track changes, and manage versioning.
- Systematically test AI model outputs for accuracy, consistency, compliance, and tone, applying rerankers, guardrails, and validation layers to ensure business rules are met.
- Collaborate with engineering and QA to promptly integrate feedback and resolve model drift or new scenarios.
- Document software and system requirements, including user stories, program functions, test cases, and steps required for new features or modifications.
- Establish systematic evaluation frameworks for AI model outputs, leveraging quantitative and qualitative benchmarks.
- Run experiments, A/B tests, and structured evaluations to compare tuning strategies and track performance metrics (accuracy, coverage, hallucination rates, latency).
Other
- Bachelor’s degree in Computer Science, Management Information Systems, Business, or related field OR equivalent work experience.
- 1+ years’ experience in software, AI/ML, or related product analysis roles.
- Experience with Agile SCRUM development processes.
- Excellent planning, organization, and interpersonal communication skills.
- Ability to work independently and collaboratively across functional groups.
