PG Forsta is seeking an experienced Staff Data Engineer to join their Unified Data Platform team to design, develop, and maintain enterprise-scale data infrastructure leveraging Azure and Databricks technologies. This role involves building robust data pipelines, optimizing data workflows, and ensuring data quality and governance across the platform to enable data-driven decision-making.
Requirements
- Advanced proficiency in Azure 5+ years (Data Lake, ADF, SQL).
- Strong expertise in Databricks (5+ years), Apache Spark (5+ years), and Delta Lake (5+ years).
- Proficient in SQL (10+ years) and Python (5+ years); familiarity with Scala is a plus.
- Strong understanding of data modeling, data governance, and metadata management.
- Knowledge of source control (Git), CI/CD, and modern DevOps practices.
- Familiarity with Power BI visualization tool.
- Experience building real-time data pipelines and streaming frameworks.
Responsibilities
- Design, build, and optimize data pipelines and workflows in Azure and Databricks, including Data Lake and SQL Database integrations.
- Implement scalable ETL/ELT frameworks using Azure Data Factory, Databricks, and Spark.
- Optimize data structures and queries for performance, reliability, and cost efficiency.
- Drive data quality and governance initiatives, including metadata management and validation frameworks.
- Collaborate with cross-functional teams to define and implement data models aligned with business and analytical requirements.
- Maintain clear documentation and enforce engineering best practices for reproducibility and maintainability.
- Support CI/CD pipeline development for data workflows using GitLab or Azure DevOps.
Other
- 7+ years of experience in data engineering, with significant hands-on work in cloud-based data platforms (Azure).
- Strong analytical and problem-solving skills.
- Proven ability to lead projects and mentor engineers.
- Excellent communication and collaboration skills.
- Exposure to machine learning integration within data engineering pipelines.
