Leidos is seeking a Databricks Data Engineer to develop/support new/existing data pipelines, and data analytics environments in Azure cloud-based data lake. As a data engineer, you will translate business requirements to data engineering solutions to support an enterprise scale Microsoft Azure based data analytics platform. You will support continued maintenance of ETL operations and development of new pipelines ensuring data quality and data management.
Requirements
- 3+ years of experience developing and designing Ingestion flows (structured, streaming, and unstructured data) using cloud platform services with data quality
- Databricks Data Engineer certification and 2+ years of experience maintaining Databricks platform and development in Spark
- Proficient at Python, Spark and R are essential.
- Knowledge and experience with data governance, including metadata management, enterprise data catalog, design standards, data quality governance, and data security.
- Experience with Agile process methodology, CI/CD automation, and cloud-based developments (Azure, AWS).
- Provision and manage infrastructure using Infrastructure-as-Code (IaC)
- Experience with the above Azure services is a plus.
Responsibilities
- Design, build, and optimize scalable data solutions using Databricks and Medallion Architecture.
- Manage ingestion routines for processing multi-terabyte datasets efficiently for multiple projects simultaneously where each project may have multiple Databricks workspaces.
- Integrate data from various structured and unstructured sources to enable high-quality business insights.
- Implement effective data management strategies to ensure data integrity, availability, and accessibility.
- Montor and support user requests, addressing platform or performance issues, cluster stability, Spark optimization, and configuration management.
- Collaborate with the team to enable advanced AI-driven analytics and data science workflows.
- Integrate with various Azure services including Azure Functions, Storage Services, Data Factory, Log Analytics, and User Management for seamless data workflows.
Other
- Ability to work directly with clients and act as front line support for requests coming in from clients.
- Clearly document and express the solution in form of architecture and interface diagrams.
- US Citizen with ability to successfully obtain government-issued Public Trust clearance
- BS degree in Computer Science or related field and 3+ years or Master’s degree with 2+ years of experience
- Certifications in Azure cloud
