Onebridge is seeking an Azure Databricks Architect to establish a strategic technology direction, architecture, and a scalable data solution using Azure Databricks, collaborating with cross-functional teams to understand business requirements, optimize data workflows, and ensure efficient processing of large datasets.
Requirements
- 12+ years of proven enterprise Data Architecture experience with expertise in designing and building data platforms and solutions in Azure Databricks.
- Extensive hands-on experience in Azure Databricks fundamentals/architecture, including setting up and optimizing Azure Databricks clusters and modules (Data Engineering, ML, and SQL warehouse).
- Experience building solutions for migrations, batches, and real-time ingestions.
- Strong experience architecting big data solutions using Data Lakes, Data Warehouses, and Data Products.
- Experience with data streaming technologies and ingestion with various versions of Informatica.
- Background with AWS components including Lambda, Kinesis/Firehose, Airflow, Redshift, Glue, PySpark, EC2, SageMaker, S3, NoSQL database, and API Gateway.
- Experience with Data Modeling, Governance, Data Management, DevOps, and MLOps.
Responsibilities
- Establish a strategic technology direction, architecture, and a scalable data solution using Azure Databricks.
- Collaborate with cross-functional teams in an Agile environment and lead efforts to understand business requirements, optimize data workflows, and ensure the efficient processing of large datasets.
- Collaborate with key stakeholders in an Agile environment to understand advanced data requirements and develop technology target states and roadmaps that align with short and long-term business goals.
- Create data models, complex ETL processes, data integrations, and CI/CD pipelines with AWS technologies.
- Define and implement best practices for data quality, data governance, and data security within Azure Databricks architecture.
- Provide knowledge on advanced concepts such as medallion architecture, DLT, unity catalog within Azure Databricks, and core AWS services/architecture best practices.
- Lead and participate in code reviews, documentation, and knowledge-sharing sessions related to Azure Databricks and AWS.
Other
- Proven track record of influencing senior leadership.
- Strong grasp of value creation and business capability models.
- Collaborate with cross-functional teams in an Agile environment.
- Highly experienced with and passionate about using Azure Databricks in large-scale technology transformations and platforms.
- Exposure to and understanding of advanced analytics techniques using ML and AI.
