Mizuho is looking to enhance its data engineering capabilities to support real-time analytics and reporting, improve data quality and governance, and automate workflows for efficient financial data processing.
Requirements
- At least 3 years of experience with Databricks and PySpark for big data processing in a financial services environment.
- 5+ years of strong programming experience in Python and advanced SQL.
- Hands-on experience with Airflow for workflow orchestration, including designing and managing complex DAGs.
- Knowledge of cloud platforms (Azure or AWS), including data lake architectures and related services such as Azure Data Lake Storage or AWS S3.
- Proven experience with Databricks and PySpark for big data processing.
- Strong programming skills in Python and advanced SQL.
- Knowledge of cloud platforms (Azure or AWS or GCP) and data lake architecture.
Responsibilities
- Design, develop, and optimize scalable data pipelines on Databricks using PySpark to process large volumes of financial data for real-time analytics and reporting.
- Implement ETL pipelines for structured and semi-structured financial data from batch and streaming processes using tools such as Apache Kafka, Airflow, and SQL.
- Implement Gold Layer transformations in Databricks for curated, high-quality datasets that support business intelligence and analytics.
- Implement DevOps best practices for data engineering workflows, including CI/CD pipelines, to ensure efficient and reliable data processing.
- Ensure data quality, governance, and compliance with financial regulations to support accurate and reliable financial reporting.
- Automate workflows using Airflow for scheduling and orchestration, improving operational efficiency.
- Optimize Spark jobs and SQL queries for performance and cost efficiency, ensuring timely and cost-effective data processing.
Other
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to deliver high-quality data solutions that meet business requirements.
- Understanding of financial domain data and regulatory requirements, with experience in handling sensitive financial information.
- Excellent problem-solving and communication skills, with the ability to work effectively in a collaborative team environment.
- Excellent problem-solving and communication skills.
- Mizuho has in place a hybrid working program, with varying opportunities for remote work depending on the nature of the role, needs of your department, as well as local laws and regulatory obligations.
