The business problem is to shape, optimize, and maintain advanced AI-driven features and user experiences within the product suite, ensuring outputs are accurate, trustworthy, and closely aligned with client needs.
Requirements
- Experience with AI/ML applications (LLMs, NLP, machine learning pipelines).
- Strong understanding of prompt engineering and generative AI interaction design.
- Basic knowledge of data modeling, relational database concepts, and SQL.
- Familiarity with model evaluation techniques and performance metrics.
- Proficiency with Python or JavaScript for prototyping and evaluation.
- Experience with Agile SCRUM development processes.
- Familiarity with RAG (retrieval-augmented generation), embeddings, MLOps, and responsible AI practices.
Responsibilities
- Design, evaluate, and refine prompts, templates, and conversation flows for reliability, cost effectiveness, and user alignment.
- Maintain prompt libraries, track changes, and manage versioning.
- Systematically test AI model outputs for accuracy, consistency, compliance, and tone, applying rerankers, guardrails, and validation layers to ensure business rules are met.
- Collaborate with engineering and QA to promptly integrate feedback and resolve model drift or new scenarios.
- Proactively gather requirements of moderate scope and complexity from diverse sources (clients, market analysis, SMEs) to drive product development and AI enhancements.
- Establish systematic evaluation frameworks for AI model outputs, leveraging quantitative and qualitative benchmarks.
- Run experiments, A/B tests, and structured evaluations to compare tuning strategies and track performance metrics (accuracy, coverage, hallucination rates, latency).
Other
- Bachelor’s degree in Computer Science, Management Information Science, Business, or related field OR equivalent work experience.
- 1+ years’ experience in software, AI/ML, or related product analysis roles.
- Excellent planning, organization, and interpersonal communication skills.
- Ability to work independently and collaboratively across functional groups.
- Experience in ERP systems, enterprise software, or complex business domains.
