SONAR/FreightWaves is looking to solve data engineering problems by hiring a Senior Data Engineer to join their fast-paced engineering team.
Requirements
- Strong working knowledge of Apache Airflow
- Experience supporting a SaaS or DaaS product, bonus points if you were creating new data products/features
- Strong in Linux environments and experience in scripting languages
- Python Expert
- Strong understanding of software best practices and associated tools.
- Experience in any major RDBMS (MySQL, Postgres, SQL Server, etc.).
- Strong SQL Skills, bonus points for having used both T-SQL and Standard SQL
Responsibilities
- Implementing ingestion pipelines, using Airflow as the orchestration platform, for consuming data from a wide variety of sources (API, SFTP, Cloud Storage Bucket, etc.).
- Implementing transformation pipelines using software engineering best practices and tools (DBT)
- Working closely with Software Engineering and DevOps to maintain reproducible infrastructure and data that serves both API-only customers and in-house SaaS products
- Defining and implementing data ingestion/transformation quality control processes using established frameworks (Pytest, DBT)
- Building pipelines that use multiple technologies and cloud environments (for example, an Airflow pipeline pulling a file from an S3 bucket and loading the data into BigQuery)
- Create and ensure data automation stability with associated monitoring tools.
- Review existing and proposed infrastructure for architectural enhancements that follow both software engineering and data analytics best practices.
Other
- Strong interpersonal skills
- Comfortable working directly with data providers, including non-technical individuals
- Bachelor's degree (not explicitly mentioned but implied)
- Ability to work in a hybrid environment with at least one day in the office per week
- No set days off Vacation policy (our team takes time off as needed with supervisor approval)
