CGI US is looking for a Senior Data Engineer (Databricks) to solve challenging business and technical problems for their clients, specifically involving migrating use cases from an on-premises Data Lake to Databricks on GCP as part of a Data Lake Modernization project.
Requirements
- 5+ years of development experience with Spark (PySpark), Python and SQL
- Extensive knowledge building data pipelines
- Hands on experience with Databricks Development
- Strong experience developing on Linux OS
- Experience with scheduling and orchestration (e.g. Databricks Workflows,airflow, prefect, control-m)
- Solid understanding of distributed systems, data structures, design principles
- Experience with Databricks Unity Catalogue
Responsibilities
- Play key role in establishing and implementing migration patterns for the Data Lake Modernization project
- Actively migrate use cases from our on-premises Data Lake to Databricks on GCP
- Collaborate with Product Management and business partners to understand case requirements and reporting
- Adhere to internal development best practices/lifecycle (e.g. Testing, Code Reviews, CI/CD, Documentation)
- Document and showcase feature designs/workflows
- Participate in team meetings and discussions around product development
- Stay up to date on industry, the latest trends and design patterns
Other
- Consultant serving local, enterprise clients
- Make an impact with both internal and client stakeholders
- Work on cutting edge technologies and cloud native development
- Inspires personal and professional growth and values your ideas
- Comfortable communicating with teams via showcases/demos
- Agile Development Methodologies (e.g. SAFe, Kanban, Scrum)
- Bachelor’s in computer science, Computer Engineering or related field
