Interplay Learning is seeking a Data Engineer to design, build, and maintain robust and scalable data pipelines and infrastructure to ensure data accuracy, accessibility, and reliability for deriving timely insights and supporting critical business operations.
Requirements
- Comfort with Python, especially for extracting data from APIs. We use dagster+dlt.
- Production experience with orchestrators like dagster or airflow, and data transformation tools like dbt.
- Preferred strong SQL and data warehouse management experience. We use BigQuery.
- Have a software engineering mindset and toolkit - git, testing, code reviews.
- Bonus points for experience with CDC pipelines.
Responsibilities
- Design and build data pipelines: Develop, construct, test, and maintain scalable data pipelines for ingesting, transforming, and loading data from various sources into our data warehouse.
- Manage data infrastructure: Oversee and optimize our data warehouse (BigQuery) and our Dagster instance, ensuring data quality, integrity, performance, and cost-effectiveness.
- Ensure data reliability and accessibility: Implement processes for data validation, error handling, and monitoring to ensure data accuracy and availability for analytical and operational needs
- Support data consumers: Collaborate with data analysts and analytics engineers to understand their data requirements and provide efficient data solutions.
- Automate data processes: Identify opportunities to automate manual data processes, improving efficiency and reducing operational overhead.
- Implement data governance: Work to establish and enforce data governance best practices, including data security, privacy, and compliance.
Other
- At least 1 year of professional experience (could be from an internship).
- A self-starter: You have a strong disposition to thrive in ambiguity, taking initiative to create clarity and forward progress.
- A great communicator: You write clear documentation and technical proposals. You are able to be precise and technical with engineers and translate technical explanations for non technical roles. You speak up when you need help.
- Collaborative team player: You value team success over ego. You will engage in spirited discourse, but can disagree and commit if the team chooses another path.
- Detail-oriented: You triple-check your numbers, and ask the clarifying questions. You have strong problem-solving and analytical skills.
