SMBC Group is looking to implement, support, and enhance its internal fraud detection platform by developing scalable data pipelines and ensuring compliance with risk management, legal, and regulatory requirements.
Requirements
- Minimum 5 years of experience in data engineering, with a focus on Databricks, PySpark, and Azure.
- Experience with CI/CD pipelines and version control using Azure DevOps or githib.
- Strong understanding of data integration, transformation, and migration strategies.
- Deep functional and technical expertise in databricks development, including strong development skills in PySpark and the Azure cloud ecosystem.
- Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar.
- Experience in developing and implementing disaster recovery strategies for cloud-based applications
Responsibilities
- Design, develop, and optimize large-scale batch data pipelines using Databricks and PySpark on the Azure cloud platform.
- Lead technical architecture and implementation of Azure-based solutions, supporting cloud migration and consolidation initiatives.
- Build and maintain ETL processes, ensuring seamless data integration and high data quality across diverse sources.
- Develop orchestration workflows using Azure Functions, Azure Data Factory (ADF), Logic Apps, and other Azure services.
- Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar
- Implement secure and scalable solutions leveraging Blob Storage, Key Vault, Managed Identities, and Azure DevOps.
- Incorporating a variety of data processing techniques—including batch and streaming workflows—while exposing and integrating APIs and external services into Databricks pipelines to enhance platform functionality and enable seamless data exchange across systems.
Other
- The ideal candidate holds a degree in Computer Science or a related field and has at least 5 years of professional experience in data engineering and cloud-based development.
- Excellent problem-solving skills and ability to resolve moderately complex technical challenges.
- Strong communication and collaboration skills;
- SMBC’s employees participate in a Hybrid workforce model that provides employees with an opportunity to work from home, as well as, from an SMBC office.
- SMBC requires that employees live within a reasonable commuting distance of their office location.
