The Commercial Bank at KeyBank is looking to identify, develop, and critically evaluate AI-powered solutions across a range of use-cases spanning sales enablement, marketing content generation, enhanced customer onboarding experiences, intelligent servicing, and novel risk assessment methodologies.
Requirements
- Expertise with traditional Machine Learning (ML)/Artificial Intelligence (AI) modeling practices, with demonstrated experience in Generative AI techniques and a deep understanding of LLM architecture, capabilities, and limitations.
- Advanced skills in tools such as SQL, Python, R, Cognos, SAS, Tableau, Excel, and Google Cloud for data manipulation and analysis.
- Hands-on work experience with statistical coding in Python, including experience with LLM libraries and frameworks (e.g., Hugging Face Transformers, LangChain, OpenAI API) and relevant data manipulation libraries (Pandas, NumPy).
- Knowledge of and ability to leverage traditional databases, cloud-based computing, and distribution computing. Familiarity with vector databases and MLOps principles specifically for managing and deploying generative AI models.
- Knowledge of financial crime regulatory requirements, technology, and data analysis best practices. Understanding of AI ethics principles, responsible AI governance frameworks, and specific regulatory considerations for AI/LLM deployments.
- Experience in developing and implementing LLM evaluation frameworks, including defining and applying relevant metrics.
- Familiarity with prompt injection vulnerabilities and mitigation strategies.
Responsibilities
- Perform a broad range of quantitative works, including designing, developing, and implementing generative AI solutions, with a strong emphasis on prompt engineering and fine-tuning of LLMs to address business needs while adhering to model risk and regulatory requirements.
- Conduct ad hoc analysis of generative model outputs to identify areas for improvement.
- Research, compile, and evaluate large sets of data, focusing on suitability for LLM training, fine-tuning, and importantly, for robust evaluation of LLM outputs. This includes creating and curating ground truth datasets and adversarial examples for performance assessment.
- Develop and maintain internal frameworks for evaluating LLM performance, including metrics for accuracy, fluency, coherence, factual correctness, toxicity, bias, and safety. Test and configure vendor-provided LLM solutions, assessing their suitability and performance against defined criteria.
- Document LLM prompt engineering strategies, model configurations, evaluation methodologies, and observed behaviors, including limitations and potential failure modes. Support model validation by providing clear documentation on how the LLM functions and the results of its evaluations.
- Employ innovative techniques to drive continuous improvements in LLM output quality, relevance, and safety. Focus on techniques such as reinforcement learning from human feedback (RLHF), prompt optimization, and bias mitigation strategies to enhance effectiveness and efficiency, e.g., improving factual accuracy and reducing harmful outputs.
- Proactively develop and build technical skills and business knowledge, particularly in Generative AI ethics, responsible AI principles, and LLM-specific risk management. Effectively collaborate with compliance, technology, and risk partners to ensure the safe and ethical deployment of AI applications.
Other
- Available for East Coast meetings, required
- Delivers clear, persuasive communication tailored to stakeholders; proactively shares relevant information and excels in high-stakes or conflict-driven conversations.
- Builds strong relationships and collaborates effectively; consults with mid-level leaders to resolve key issues and influence outcomes.
- Leverages deep banking and financial insight to drive data-informed strategies aligned with business goals and market dynamics
- Bachelor's Degree in business, finance, MIS, analytics/data science, engineering, or related field, required
