Intuit Credit Karma is seeking a Staff Data Engineer to lead the data architecture and engineering strategy for their experimentation platform. The goal is to build and scale data systems that provide reliable, consistent, and timely experimentation insights across the company, supporting thousands of metrics and diverse statistical methodologies.
Requirements
- Proficiency in Scala, Python, and SQL.
- Demonstrated success building and maintaining large-scale data pipelines using technologies such as Spark, Flink, Google Dataflow, BigQuery, or Airflow/Composer.
- Familiarity with Python libraries for statistical analysis (e.g., Statsmodels, SciPy).
- Proven expertise in A/B testing methodologies and statistical concepts.
- Previous experience building analytics pipelines at experimentation platform at large-scale tech companies serving millions of users or vendors (e.g., Optimizely, Statsig, LaunchDarkly)
- Knowledge of heterogeneous treatment effects and advanced statistical modeling techniques for experimentation.
- Experience with adaptive experimentation or Bayesian optimization methods.
Responsibilities
- Define Technical Strategy: Provide the roadmap and architecture for the experimentation platform's infrastructure, ensuring alignment with business objectives and adherence to industry best practices.
- Develop Near Real-Time Systems: Lead critical initiatives to build our next-generation near real-time ecosystem, to enhance near real-time observability and alerting, leveraging Scala, Pub/Sub, Akka, and Dataflow on Google Cloud.
- Build Scalable Pipelines: Architect and maintain large-scale batch data pipelines using Google Dataflow, BigQuery, and Airflow/Cloud Composer to handle high-volume, batch data processing.
- Develop Core Capabilities: Enhance the experimentation platform with new capabilities such as experiment targeting and localized assignments at scale to reduce latency and improve developer experience.
- Optimize Data Infrastructure: Drive efficiency and performance improvements across experimentation pipelines, frameworks, and query layers. Evaluate trade-offs in system design, balancing speed, scalability, cost, and accuracy.
- Stay Current with Industry Trends: Research, evaluate, and integrate the latest advancements in experimentation methods, data analysis techniques, and cloud-based technologies to continually improve the platform.
- Collaborate on Experiment Analysis: Partner with marketers, analysts, and data scientists to build infrastructure that supports thousands of metrics and various statistical methods (e.g., t-tests, sequential testing, Bayesian analysis).
Other
- 10+ years in software engineering, with a focus on data engineering and data architecture.
- Deep understanding of software development lifecycle best practices, including agile methodologies.
- Excellent communication, collaboration, and stakeholder management skills.
- Proven ability to lead complex projects and mentor engineering teams.
- Mentor and Guide: Provide technical leadership and support to junior engineers, fostering a culture of continuous learning and professional growth.
