Designing, building, and maintaining scalable data pipelines that transform IoT and time-series data into actionable insights to improve operational efficiency and drive business outcomes.
Requirements
- 5+ years of experience building end-to-end analytics solutions, including real-time and batch data pipelines for IoT or time-series data.
- 2+ years of experience operationalizing analytics pipelines in containerized environments (Docker, Kubernetes).
- 2+ years of AI/ML pipeline development experience with frameworks such as TensorFlow, PyTorch, MLFlow, or SparkML.
- Hands-on experience with Apache Flink, Spark, Structured Streaming, Akka, or Kafka.
- Expertise in cloud-based platforms, particularly Databricks, for scalable batch processing.
- Ability to thrive in ambiguous environments while maintaining high standards of data quality and performance.
Responsibilities
- Design and implement scalable data pipelines for IoT telemetry and time-series data using Apache Flink, Spark, and Databricks.
- Build and maintain time-series databases and data models using PostgreSQL and TimescaleDB.
- Integrate data governance practices, including metadata management, lineage tracking, and compliance within a cloud-based data lake ecosystem.
- Optimize batch and streaming pipelines through performance tuning, partitioning, caching, and adaptive query execution.
- Develop Infrastructure as Code (IaC) solutions using Terraform and Azure Bicep to provision and manage cloud resources across multi-cloud platforms.
- Collaborate with data scientists and engineers to deploy AI/ML solutions, including predictive analytics, early alerts, and prescriptive recommendations.
- Write clean, scalable, and highly efficient code in Python, Java, PySpark, Scala, and R.
Other
- Bachelor’s degree in Computer Science, Statistics, Mathematics, or related field.
- Strong problem-solving, critical thinking, and communication skills, with the ability to work across organizational boundaries.
- Demonstrated initiative to tackle complex problems at scale and operationalize solutions.
