Best Egg is looking to build and scale the core data and feature engineering pipelines that power machine learning and AI at Best Egg, enabling data scientists to quickly prototype, train, and deploy models with consistent, reliable, and high-quality features.
Requirements
- Advanced SQL skills, experience with Python dataframes (e.g., Polars, pandas, Narwhals), and a deep understanding of data modeling and feature engineering best practices.
- Significant experience with Snowflake, BigQuery, Redshift, or Databricks in production settings.
- Ability to design Python data pipelines (DAGs) that work seamlessly across batch and real-time contexts, including concepts like incremental processing and backfills.
- Solid Python development skills (concurrency, async, API design) and experience with FastAPI (or similar) for building data services.
- Comfort in quickly investigating complex issues across SQL, Python, and infrastructure layers.
- Working experience with Docker, Kubernetes, CI/CD workflows, and infrastructure-as-code (Terraform/CloudFormation).
Responsibilities
- Design and implement SQL- and Python-based pipelines (using dataframes such as Polars/Narwhals) that support both backfills for training and low-latency, real-time serving.
- Collaborate with data scientists to design, build, and maintain new features from complex time series data sources, ensuring they are reusable, well-documented, and consistent across environments.
- Help data scientists troubleshoot complex SQL queries, debug feature outputs, and optimize queries for performance.
- Quickly diagnose and resolve platform issues spanning Python services (FastAPI), Snowflake queries, Kubernetes services, and real-time pipelines.
- Deliver high-quality abstractions, tools, and libraries that simplify feature development and improve data scientist workflows.
- Monitor, profile, and optimize data pipelines and feature services for throughput, latency, and cost efficiency.
Other
- 5+ years in data engineering, backend engineering, ML engineering, or related software development roles with a proven track record of building and maintaining large-scale data systems.
- Ability to partner effectively with data scientists, analysts, and engineers while promoting best practices in feature engineering and data platform design.
- In addition to semi-monthly salary payments, this position is also eligible for an annual incentive bonus based on individual and company performance.
- The yearly incentive bonus target is 20%
- Best Egg celebrates diversity and equal opportunity. We are committed to building a team that represents a variety of backgrounds, perspectives, and skills.
