SMBC Group is looking to implement, support, and enhance its internal fraud detection platform by developing scalable data pipelines and integrating batch processing methods to align with the bank's risk management, legal, and regulatory requirements for fraud detection and prevention.
Requirements
- deep functional and technical expertise in databricks development
- strong development skills in PySpark
- Azure cloud ecosystem
- Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar
- Minimum 5 years of experience in data engineering, with a focus on Databricks, PySpark, and Azure.
- Strong understanding of data integration, transformation, and migration strategies.
- Experience with CI/CD pipelines and version control using Azure DevOps or githib.
Responsibilities
- Design, develop, and optimize large-scale batch data pipelines using Databricks and PySpark on the Azure cloud platform.
- Lead technical architecture and implementation of Azure-based solutions, supporting cloud migration and consolidation initiatives.
- Build and maintain ETL processes, ensuring seamless data integration and high data quality across diverse sources.
- Develop orchestration workflows using Azure Functions, Azure Data Factory (ADF), Logic Apps, and other Azure services.
- Proven expertise in designing and managing CI/CD pipelines using tools such as Azure DevOps, GitHub, or similar
- Implement secure and scalable solutions leveraging Blob Storage, Key Vault, Managed Identities, and Azure DevOps.
- Provide technical guidance and support for architectural decisions and platform enhancements.
Other
- work closely with business units and support teams to deliver the initial application, system enhancements, perform upgrades, and provide on-call user support.
- Own end-end project delivery, working closely with business stakeholders, IT teams, and third-party vendors.
- Review and contribute to core code changes, ensuring best practices and supporting production deployments.
- Bachelor’s degree in Computer Science, Information Systems, or a related technical field.
- Excellent problem-solving skills and ability to resolve moderately complex technical challenges.
