Drives AI/ML and LLM technology architecture that enables data scientists in the development and deployment of AI Algorithms, allowing data scientists to focus on model development while the architecture team takes care of all engineering needs for a self-sustained AI Platform.
Requirements
- Demonstrated Expertise (“DE”) architecting, building, and deploying data pipelines on AWS cloud using AWS services (IAM, Glue,S3, Lambda, and EKS), Jenkins, Python, Airflow and Dynamo DB.
- DE implementing Cloud-based ML or Data Engineering pipelines using Amazon Web Services (AWS)and using EMR, RDS PostgreSQL, Redshift, Athena, and Step functions.
- DE developing real time streaming and batch data pipelines in support of data analytics, Machine Learning (ML), and Artificial Intelligence (AI) processing using Kafka, Apache Spark, Python, and AWS.
- DE building and maintaining data engineering pipelines or Al pipelines using Jenkins, Terraform, SQS, API Gateway, and Dynamo DB.
Responsibilities
- Drives AI/ML and LLM technology architecture that enables data scientists in the development and deployment of AI Algorithms
- Architects and builds distributed data intensive applications.
- Commercializes ChatGPT or GPT3/4 for business applications.
- Optimizes use of cloud services and efficiency drive, running PoCs on new cloud services as applicable to platform and helping to increase the speed of deployment for all data science use cases.
- Architecting, building, deploying, and monitoring real-time and batch pipelines for data engineering.
- Architecting, building, and deploying data pipelines on AWS cloud using AWS services (IAM, Glue, S3, Lambda, and EKS), Jenkins, Python, Airflow and Dynamo DB.
- Implementing Cloud-based ML or Data Engineering pipelines using Amazon Web Services (AWS) and using EMR, RDS PostgreSQL, Redshift, Athena, and Step functions.
Other
- Partners with the Product Owners, Product Area Leaders and Chapter leaders to define a long-term strategy for AI Platform.
- Partners with architecture leaders from across brokerage and partner business units and the enterprise to help influence the broader AI/ML technology community.
- Works closely with the Data Science community, Data Engineering team, the enterprise data community, the analysts, enterprise cloud and security teams.
- Meets with department heads, managers, supervisors, vendors, and others, to solicit cooperation and resolve problems.
- Reviews project plans to plan and coordinate project activity.
