Visa's Technology Organization is looking to design, scale, and optimize its large-scale data engineering and AI/ML infrastructure to power mission-critical products and transform its post-purchase ecosystem into a data-first, AI-powered platform.
Requirements
- Deep experience in Hadoop ecosystem, Apache Spark, Databricks, Airflow, Python, SQL, and Scala
- Strong knowledge of AWS/Azure cloud data services, including S3, Glue, Synapse, Redshift, and Delta Lake.
- Proficiency with ETL frameworks, Kafka, Airflow, and Kubernetes-based containerized environments.
- Hands-on experience designing REST APIs, data ingestion frameworks, and metadata management systems.
- Familiarity with MCP tools and clients, RAG pipelines, and Lang Chain/Lang Graph for generative AI integration.
- Expertise in data modeling, dimensional schema design, and performance optimization for analytics and ML workloads.
- Data Systems: Hadoop, Hive, Spark, Databricks, Snowflake, Delta Lake, Airflow
Responsibilities
- Architect and Lead Data Solutions: Design and implement large-scale, distributed data systems integrating Hadoop, Apache Spark, Databricks, and cloud-native (AWS/Azure) services.
- Build Enterprise Data Pipelines: Develop robust ETL and streaming pipelines that handle large volumes of transactional and behavioral data, supporting machine learning and analytics workloads.
- Data Warehouse Modernization: Lead modernization of the Hadoop-based data warehouse, ensuring scalability, data quality, lineage, and governance standards are met.
- AI/ML Data Infrastructure: Collaborate with AI scientists to deliver data models optimized for ML pipelines, feature engineering, and model training using frameworks like LangChain, LangGraph, and MCP.
- Insight Generation: Design and build reusable frameworks to extract, transform, and surface insights from structured, semi-structured, and unstructured data sources (e.g., JSON, logs, events).
- Proof of Concept & Innovation: Lead POC initiatives to evaluate emerging technologies (Delta Lake, Iceberg, Kafka, Arrow, etc.) and translate findings into scalable production systems.
- Security & Compliance: Collaborate with InfoSec to ensure end-to-end data encryption, secure access, and compliance with Visa’s global data privacy standards.
Other
- This is a hybrid position. Expectation of days in the office will be confirmed by your Hiring Manager.
- Energy and Experience: A growth mindset that is curious and passionate about technologies and enjoys challenging projects on a global scale
- Challenge the Status Quo: Comfort in pushing the boundaries, ‘hacking’ beyond traditional solutions
- Learner: Constant drive to learn new technologies
- Partnership: Experience collaborating with Product, Test, Dev-ops, and Agile/Scrum teams
