DataSF is seeking a Senior Data Engineer to join their growing team to design, build, and maintain the City's data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs, contributing directly to San Francisco's commitment to efficient, equitable, and ethical service delivery.
Requirements
- Hands-on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
- Demonstrated expertise in writing advanced, performant SQL, and using tools like dbt for SQL-based data transformation and modeling.
- Strong programming skills in Python (with libraries like pandas, PySpark) for data processing and automation.
- Proficiency with an Infrastructure as Code tool, with a preference for Terraform.
- Experience building and deploying data pipelines using orchestration tools like Azure Data Factory, Airflow, Dagster, or similar technologies.
- Deep understanding of data warehousing concepts, data modeling, and modern ELT principles.
- Experience with real-time data streaming technologies (e.g., Kafka, Kinesis, Snowpipe).
Responsibilities
- Manage our central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization.
- Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources.
- Design infrastructure to support both streaming and batch processes, and both structured and unstructured data sources.
- Use Terraform to define, deploy, and manage data infrastructure, ensuring our pipelines are reproducible, version-controlled, and production-ready.
- Champion and implement best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing.
- Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability.
Other
- Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.
- Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders in diverse domains to understand data requirements and build solutions that meet their needs.
- Excellent verbal and written communication skills, including the ability to explain technical concepts to non-technical stakeholders.
- A collaborative mindset with enthusiasm to work across diverse, cross-functional teams.
- Commitment to equity, transparency, and ethical data use.
