The UT Data Hub needs to improve university outcomes and advance the UT mission by increasing the usability and value of institutional data. This role will lead senior data engineers and data engineers to create complex data pipelines within UT’s cloud data ecosystem to support academic and administrative needs, and help build and run a modern data hub for advanced data-driven decision making.
Requirements
- 5+ years of experience designing, implementing, and optimizing complex, production-grade data pipelines or enterprise-scale data platforms.
- 5 years of experience in cloud-based data engineering using Databricks and Amazon Web Services (AWS) (e.g., Glue, S3, Lambda, Redshift).
- Expertise in Python, PySpark, and SQL, with strong understanding of data modeling, stored procedures, and scalable data transformations.
- Proven experience architecting and implementing ETL/ELT solutions across relational, non-relational, and lakehouse environments (e.g., Delta Lake, Parquet, or Iceberg).
- Experience designing and managing CI/CD pipelines and infrastructure as code (IaC) using tools such as Databricks Repos, CDK, Terraform, or GitHub Actions.
- Demonstrated knowledge of test-driven development (TDD) and data quality frameworks, ensuring reliability and reproducibility across data workflows.
- Deep understanding of data governance, security, and compliance standards in cloud environments.
Responsibilities
- Design, architect, and deliver production-grade, scalable data pipelines and AI-ready data platforms using Databricks, AWS cloud-native services and modern data engineering frameworks.
- Lead end-to-end implementation of lakehouse data pipelines, ensuring performance, reliability, and cost efficiency.
- Champion industry best practices for data engineering.
- Conduct and participate in peer code reviews to maintain code quality and consistency across the team.
- Proactively identify and resolve bottlenecks in data ingestion, transformation, and orchestration processes using Databricks Delta Live Tables, Spark optimization techniques, and workflow automation.
- Implement systems for data quality, observability, governance, and compliance using tools such as Unity Catalog, Delta Lake, and data validation frameworks.
- Lead technical knowledge-sharing sessions on topics such as AI/ML integration, data lakehouse architecture, and emerging data technologies.
Other
- 3+ years of experience managing or leading teams of data and/or software engineers, including mentorship, performance management, and project delivery.
- Proven ability to communicate complex technical concepts clearly to both technical and non-technical audiences.
- Experience supervising, mentoring, and guiding junior team members on technical and professional development.
- Must be authorized to work in the United States on a full-time basis for any employer without sponsorship.
- This position requires you to maintain internet service and a mobile phone with voice and data plans to be used when required for work.
