YipitData is looking to build and scale its vendor universe, which is the database of companies it tracks and analyzes. This requires designing and maintaining large-scale data pipelines, defining best practices for ETL, and integrating emerging technologies to keep the platform on the cutting edge.
Requirements
- Proficiency with PySpark and Databricks for processing and scaling large datasets.
- Hands-on experience with Airflow for pipeline orchestration (Dagster/dbt a plus).
- Strong expertise in SQL and distributed data systems.
- Experience leveraging AI/ML models, vector search, or Elasticsearch to enhance data pipelines.
- Experience building a reliable and efficient data pipelines
Responsibilities
- Own the design, build, and optimization of end-to-end data pipelines that power our vendor universe.
- Establish and enforce best practices in data modeling, orchestration, and system reliability.
- Collaborate with product, engineering, and business stakeholders to translate requirements into robust, scalable data solutions.
- Work extensively with Databricks and Airflow for large-scale data processing and orchestration.
- Troubleshoot and resolve complex pipeline issues to ensure reliability and performance.
- Contribute to the team’s technical strategy, helping drive improvements in scalability, performance, and efficiency.
- Lead, mentor, and support engineers through challenges, code reviews, and project execution.
Other
- 6+ years of professional experience in Data Engineering or equivalent technical roles (e.g., data architecture, big data development, or ETL engineering).
- 2+ years of managerial experience, including mentoring, team leadership, and supporting delivery.
- Proven track record of delivering in fast-paced, deadline-driven environments with minimal oversight.
- Strong problem-solving skills and ability to translate business needs into scalable technical solutions.
- Excellent communication and collaboration skills with both technical and non-technical stakeholders.
- This is a remote-friendly opportunity that can sit in NYC (where our headquarter is located), one of our office hubs (Austin, Miami, Denver or Mountain View,), or anywhere else in the United States.
- Please note that for this position, we are not able to consider candidates who currently or in the future will require visa sponsorship.
