SONAR/FreightWaves needs a Senior Data Engineer to build and maintain data pipelines and infrastructure that support their logistics and supply chain data and analytics products, empowering risk management and market insights.
Requirements
- Strong working knowledge of Apache Airflow
- Strong in Linux environments and experience in scripting languages
- Python Expert
- Strong understanding of software best practices and associated tools.
- Experience in any major RDBMS (MySQL, Postgres, SQL Server, etc.).
- Strong SQL Skills, bonus points for having used both T-SQL and Standard SQL
- Experience with NoSQL (Elasticsearch, MongoDB, etc.)
Responsibilities
- Implementing ingestion pipelines, using Airflow as the orchestration platform, for consuming data from a wide variety of sources (API, SFTP, Cloud Storage Bucket, etc.).
- Implementing transformation pipelines using software engineering best practices and tools (DBT)
- Working closely with Software Engineering and DevOps to maintain reproducible infrastructure and data that serves both API-only customers and in-house SaaS products
- Defining and implementing data ingestion/transformation quality control processes using established frameworks (Pytest, DBT)
- Building pipelines that use multiple technologies and cloud environments (for example, an Airflow pipeline pulling a file from an S3 bucket and loading the data into BigQuery)
- Create and ensure data automation stability with associated monitoring tools.
- Review existing and proposed infrastructure for architectural enhancements that follow both software engineering and data analytics best practices.
Other
- This position is fully remote.
- Must RESIDE in the United States and be eligible to work within the US.
- Strong interpersonal skills
- Comfortable working directly with data providers, including non-technical individuals
- Curious, tenacious, and team-oriented
