The company is looking to solve problems related to estimation practices, data ingestion processes, and DevOps/CI/CD pipelines to improve the realism of forecasts, standardize onboarding of new data sources, and speed up delivery without sacrificing quality.
Requirements
- 8–10 years in data engineering with strong exposure to enterprise-scale data warehouses and analytics platforms.
- Hands-on expertise with Hadoop/Spark, Databricks (PySpark, Delta Lake), Snowflake, and AWS data services.
- Demonstrated experience in Agile estimation techniques (velocity management, story points, effort estimation models).
- Ability to correlate delivery estimates with actuals and recommend right-sizing approaches for teams and projects.
- Proven track record of designing scalable ingestion frameworks (e.g., metadata-driven ETL/ELT, reusable connectors, automation accelerators).
- Strong SQL and Python development skills; experience with orchestration (Airflow, dbt) and CI/CD for data pipelines.
Responsibilities
- Validate and rationalize estimation practices (story points, t-shirt sizing, function points) by aligning team size, velocity, and backlog management to actual delivery outcomes.
- Develop a framework/checkpoints for estimation validation to ensure forecasts for new data products are realistic and do not lead to unchecked growth in headcount.
- Analyze existing data ingestion processes and recommend standardized, reusable patterns for onboarding new data sources (e.g., metadata-driven pipelines, automation, ingestion templates) that reduce one-off solutions.
- Assess DevOps and CI/CD pipelines for bottlenecks and propose improvements that speed delivery without sacrificing quality.
- Collaborate with the Solution Architect on updating the current-state architecture and drafting future-state recommendations (integration, scalability, accelerators).
- Provide technical inputs to the Project Estimation Validation Report (current practices, gap analysis, skills assessment, accuracy improvements).
- Recommend best in class estimation discipline, backlog predictability, and scalable data engineering practices.
Other
- Bachelor Degree in relevant field with 6- 8 years of relevant experience
- Consulting and communication skills to explain estimation methodology, team capacity modeling, and ingestion strategies to both technical and executive stakeholders.
