People Data Labs is seeking to solve complex data engineering and data science problems by building and scaling innovative, compliant data solutions. They aim to provide the best people and company data by integrating thousands of compliantly sourced datasets into a single, developer-friendly source of truth.
Requirements
- 4-6+ years of industry experience with clear examples of strategic technical problem-solving and implementation
- Strong software development fundamentals.
- Experience with Python
- Expertise with Apache Spark (Java, Scala, and/or Python-based)
- Experience with SQL
- Experience building scalable data processing systems (e.g., cleaning, transformation) from the ground up.
- Experience using developer-oriented data pipeline and workflow orchestration (e.g., Airflow (preferred), dbt, dagster or similar)
Responsibilities
- Build infrastructure for ingestion, transformation, and loading an exponentially increasing volume of data from a variety of sources using Spark, SQL, AWS, and Databricks
- Building an organic entity resolution framework capable of correctly merging hundreds of billions of individual entities into a number of clean, consumable datasets.
- Developing CI/CD pipelines and anomaly detection systems capable of continuously improving the quality of data we're pushing into production.
- Dreaming up solutions to largely undefined data engineering and data science problems.
Other
- Balance high ownership and autonomy with a strong ability to collaborate
- Work effectively remotely (able to be proactive about managing blockers, proactive on reaching out and asking questions, and participating in team activities)
- Demonstrate strong written communication skills on Slack/Chat and in documents
- Exhibt experience in writing data design docs (pipeline design, dataflow, schema design)
- Scope and breakdown projects, communicate and collaborate progress and blockers effectively with your manager, team, and stakeholders
