Visa is looking to solve the problem of designing, developing, and optimizing large-scale data platforms and cloud-based analytics environments to support analytics, machine learning, and real-time data needs.
Requirements
- Advanced expertise in building and optimizing large-scale distributed data systems using Hadoop, Spark, and modern lakehouse architectures.
- Strong programming proficiency in PySpark, Scala, and Python with experience implementing scalable, production-grade data applications.
- Deep experience designing and tuning RDBMS, NoSQL, and distributed SQL systems.
- Mastery of SQL and distributed query engines such as Presto, Trino, Hive, and SparkSQL.
- Strong knowledge of data modeling, ETL/ELT design, and data warehousing methodologies.
- Proven experience architecting and operating data solutions on AWS, GCP, and Azure, including cloud data lakes, orchestration tools, and cost-effective storage/compute designs.
- Advanced proficiency in Databricks, including: Building and optimizing notebooks and production jobs, Delta Lake design and optimization, Cluster configuration and workspace administration, CI/CD integration for data workloads, Performance tuning for large distributed jobs.
Responsibilities
- Lead the architecture and delivery of large-scale, high-performance data pipelines and processing frameworks across Hadoop and multi-cloud environments.
- Design scalable data models, lakehouse structures, and distributed data processing solutions that support analytics, machine learning, and real-time data needs.
- Provide technical leadership to Senior and Staff Data Engineers, conducting design reviews, guiding implementation decisions, and ensuring engineering excellence.
- Develop and improve engineering best practices for data governance, quality, observability, testing, and cloud resource optimization.
- Drive adoption of cloud-native data technologies, automation frameworks, and reusable components that improve development velocity and system reliability.
- Lead complex data modernization efforts, including cloud migration, data lake/lakehouse consolidation, and performance optimization of critical pipelines.
- Evaluate new tools and technologies, influencing platform evolution within the scope of assigned domains or product areas.
Other
- This is a hybrid position. Expectation of days in the office will be confirmed by your Hiring Manager.
- Collaborate with product, analytics, and platform teams to ensure alignment on data strategy and architectural roadmaps.
- Mentor engineers at all levels, providing technical coaching and fostering a culture of continuous improvement.
- This position requires travel 5-10% of the time.
- Strong problem-solving skills with the ability to troubleshoot complex data and performance issues.
