Sam's Club is looking to centralize its testing and measurement processes by building a unified experimentation and measurement platform and launching self-service tools to empower associates to conduct high-quality experiments independently, ultimately driving measurable business impact and improving decision-making.
Requirements
- Deep experience designing and validating RCTs and quasi-experiments (e.g., diff-in-diff, synthetic controls, matching/stratification), power/MDE analysis, sequential testing control, variance reduction (e.g., CUPED), and heterogeneous effect analysis.
- Proven track record building or scaling experimentation/measurement platforms (assignment, identity/bucketing, telemetry, metric computation layers, registry, readouts, and observability), ideally in high-scale product or retail environments.
- Deep proficiency in Python for machine learning and statistical modeling, with strong experience using ML frameworks (e.g., TensorFlow, PyTorch, Scikit-learn) and data manipulation libraries (e.g., Pandas, NumPy).
- Skilled in SQL for feature engineering and large-scale data extraction.
- Committed to production-grade practices including code quality, automated testing, reproducibility, and model lifecycle management (MLOps).
- Experience with in-club/physical retail measurement (traffic normalization, seasonality/weather controls, club clustering) is highly preferred.
- Data science, machine learning, optimization models, PhD in Machine Learning, Computer Science, Information Technology, Operations Research, Statistics, Applied Mathematics, Econometrics, Successful completion of one or more assessments in Python, Spark, Scala, or R, Using open source frameworks (for example, scikit learn, tensorflow, torch)
Responsibilities
- Lead the centralization of testing and measurement at Sam’s Club by building a single experimentation and measurement platform and launching self-service tools that empower associates to run high-quality experiments independently.
- Establish scientific standards, governance, and guardrails that make results trustworthy and repeatable.
- Translate complex data into clear, actionable decisions.
- Inventory current experimentation practices across teams; map data sources, metrics, and decision paths; and publish a Unified Testing & Measurement Playbook (taxonomy, KPI catalog, experiment types, power/MDE guidance, guardrails, and approval paths).
- Define architecture and build-versus-buy strategy for an enterprise testing platform (assignment service, telemetry, experiment registry, metrics pipeline, results service, observability, and audit trails).
- Ship a guided, self-service tool enabling ~80% of standard experiments without DS support (A/A checks, sizing calculators, pre-registration, guardrails, and standardized readouts).
- Stand up robust in-club measurement for prioritized initiatives (e.g., signage, layouts, checkout prompts, labor/process changes) using appropriate designs (RCTs, matched controls, diff-in-diff, synthetic control).
Other
- Mentor a high-performing team, and partner across Product, Engineering, and the business to drive measurable business impact—while reserving your team’s deep expertise for the most complex use cases.
- Secure cross-functional alignment.
- Train product/ops partners and stand up community of practice.
- Produce a quarterly experimentation portfolio review quantifying incremental sales, margin, and member engagement; prioritize the roadmap to maximize ROI.
- Hire, coach, and retain a blended team (data scientists, measurement scientists, and platform/analytics engineers). Establish high-quality code, review, and documentation standards; promote a culture of healthy debate, learning, and outcomes.
- Ability to translate noisy signals into clear decisions; influence senior leaders with concise narratives that connect experiments to member outcomes and financial impact.
- Experience building and mentoring teams, driving cross-functional adoption, and moving organizations from fragmented practices to consistent standards and self-service at scale.
