The Cloud Data Engineer collaborates with cross-functional teams to deliver efficient data engineering processes, data solutions, and operations support.
Requirements
- Proven experience as a Cloud Data Engineer with expertise in AWS services.
- Strong background in designing and implementing ETL processes.
- Familiarity with data modeling, database design, and optimization techniques.
- AWS certifications in relevant areas.
- Experience with Apache Nifi, Apache Nifi Registry
- Knowledge of data warehousing concepts and practices.
- Familiarity with DevOps practices in a cloud environment.
Responsibilities
- Design, develop, and maintain scalable cloud-based data architectures on AWS and Databricks.
- Implement efficient ETL/ELT processes using Databricks notebooks, Delta Lake, and Apache Spark.
- Utilize AWS services such as S3, Glue, Redshift, and EMR, and integrate with Databricks Workspaces for unified data processing.
- Ensure data security, integrity, and compliance with industry standards and regulations, including role-based access control (RBAC) in Databricks.
- Troubleshoot and optimize existing ETL processes for performance and reliability using Databricks performance tuning techniques.
- Stay updated on emerging cloud technologies and recommend improvements to enhance data engineering capabilities, including Databricks Lakehouse architecture.
- Collaborate with data scientists and analysts to provide accessible and high-quality data for analytics and reporting via Databricks SQL and Unity Catalog.
Other
- Support clear communications both within the organization and to business partners around any data initiatives, planning, and problem resolutions, including Databricks workspace collaboration.
- Work closely with our partners and internal staff to address any data-related concerns promptly.
- Ability to work collaboratively in a team environment.
- Strong communication skills to articulate technical concepts to non-technical stakeholders.
- Ability to interpret a variety of instructions furnished in written, oral, diagram or schedule form.
