Open Lending is seeking to drive the continued evolution of their modern data platform leveraging Databricks, Unity Catalog, Lakehouse architecture, as well as assisting in the continued support and development of their data warehouse.
Requirements
- 3+ years of hands-on experience with Azure Databricks, including PySpark, Delta Lake, Unity Catalog, and Databricks Workflows.
- 2+ years designing and maintaining data pipelines using Azure services such as Data Factory, Event Hubs, Key Vault, and Logic Apps.
- Expertise in Python/PySpark for large-scale transformations, orchestration, and automation.
- Strong knowledge of distributed data systems and Lakehouse design principles.
- Experience with streaming architectures (Spark Streaming, Kafka, Event Hubs).
- Proficiency with SQL and familiarity with document databases.
- Working understanding of Unity Catalog governance, lineage, and security controls.
Responsibilities
- Design, build, and maintain scalable data ingestion and transformation pipelines using Databricks (PySpark, Delta Lake, Workflows) and Azure Data Factory.
- Implement Lakehouse architecture best practices with Delta Lake for performance, reliability, and ACID compliance.
- Utilize Unity Catalog for centralized data governance, lineage tracking, and fine-grained access controls.
- Develop and maintain data validation frameworks within Databricks to ensure accuracy, completeness, and compliance across all data domains.
- Build both batch and streaming data pipelines using Spark Structured Streaming, Kafka, or Azure Event Hubs.
- Drive performance tuning, cost optimization, and observability of data pipelines and compute clusters
- Improve resiliency and maintainability through testing, alerting, and automation with Databricks workflows and Lakehouse monitoring.
Other
- Bachelor’s degree in Computer Science, Information Technology, or related field.
- 5+ years of experience in Data Engineering, ETL, or Data Warehousing.
- Excellent collaboration, communication, and problem-solving skills.
- Prolonged periods sitting at a desk and working on a computer.
- Must be able to lift 15 pounds at times.
