Transforming and optimizing data architecture to support WTW's data-driven strategy.
Requirements
- Extensive experience as a Data Engineer, with a focus on Python programming in Spark/Databricks environments.
- Proven expertise in designing and optimizing large-scale data pipelines, ETL processes, and data lakes.
- Strong background in cloud platforms, ideally Azure, with hands-on experience managing cloud-based data infrastructure.
- Demonstrated ability to convert legacy systems into modern Spark/Databricks platforms, improving efficiency and performance.
- Excellent problem-solving skills with the ability to troubleshoot and resolve complex data issues.
Responsibilities
- Develop and maintain scalable data pipelines and systems using Scala, Python, Spark, and Databricks to process and manage large-scale datasets.
- Design and implement unified data models with enterprise-wide consistency in key naming conventions, data types, and transformation functions.
- Optimize operations in Spark for handling large, unstructured, and nested datasets by implementing efficient data structuring and processing techniques.
- Collaborate with data science and analytics teams to create and refine data processes that enhance decision-making and align with business goals.
- Lead initiatives to streamline and enhance data quality monitoring systems.
Other
- Strong communication and collaboration skills, with the ability to work effectively in a remote team environment.
- Employment-based non-immigrant visa sponsorship and/or assistance is not offered for this specific job opportunity.
- The majority of our colleagues work in a ”hybrid” style, with a mix of remote, in-person and in-office interactions dependent on the needs of the team, role and clients.
- Pursuant to the San Francisco Fair Chance Ordinance and Los Angeles County Fair Chance Ordinance for Employers, we will consider for employment qualified applicants with arrest and conviction records.
- EOE, including disability/vets
