Drive the evolution of our real-time data platform, enabling real-time insights that directly impact business outcomes.
Requirements
- 8+ years of experience in data engineering, with at least 3+ years in real-time or streaming systems.
- Deep expertise with Kafka (or equivalent message brokers), including schema registry, partitioning strategies, and consumer group management.
- Hands-on experience with stream processing frameworks (e.g., Apache Flink, Spark Structured Streaming, Kafka Streams).
- Strong proficiency in Python, Scala, or Java, and familiarity with data pipeline orchestration tools (Airflow, Dagster, etc.).
- Solid understanding of distributed systems, networking, and storage.
- Experience with cloud platforms (AWS, GCP, or Azure) and managed streaming services (Kinesis, Pub/Sub, etc.).
- Familiarity with data modeling, lakehouse architectures, and batch + streaming integration.
Responsibilities
- Architect and lead the design and implementation of scalable, fault-tolerant streaming data pipelines using platforms such as Apache Kafka, Flink, Spark Structured Streaming, or Kinesis.
- Evolve the real-time data platform to support new business use cases, ensuring low-latency data delivery and high reliability.
- Partner with data scientists, product engineers, and platform teams to design data models and streaming APIs that enable real-time insights and decision-making.
- Drive data quality, observability, and governance across streaming systems, including schema management, lineage, and monitoring.
- Optimize pipeline performance, reliability, and cost across multiple environments (cloud and on-prem).
- Lead code reviews, architecture reviews, and technical mentoring for other engineers.
- Establish and champion best practices for streaming data engineering, deployment automation, and CI/CD for data systems.
Other
- A strong sense of ownership and curiosity for optimizing data flow in complex environments.
- Ability to think strategically while delivering practical, scalable solutions.
- Leadership in driving cross-team collaboration and influencing architectural direction.
- Passion for helping teams move from batch-oriented thinking to event-driven design.
- REMOTE
