Effectual is seeking a Senior Data Engineer with specialized expertise in data streaming technologies to build and maintain high-performance data streaming architectures for real-time data processing and analytics.
Requirements
- Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools
- Proven experience with AWS MSK and Amazon Kinesis services in production environments
- Strong background in real-time data processing and stream analytics
- Streaming Technologies: Apache Kafka, Kafka Connect, Kafka Streams, Amazon MSK, Amazon Kinesis (Data Streams, Data Firehose, Analytics)
- Programming Languages: Proficient in Python, Java, and Scala for streaming applications
- Stream Processing Frameworks: Apache Spark Streaming, Apache Flink, AWS Lambda for stream processing
- AWS Services: Deep knowledge of AWS streaming and analytics services (MSK, Kinesis, Lambda, EMR, Glue)
Responsibilities
- Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis
- Develop real-time data pipelines that handle high-volume, high-velocity data streams
- Implement event-driven architectures and microservices patterns for streaming data processing
- Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments
- Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions
- Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda
- Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases
Other
- 7+ years of experience in the data engineering field with significant streaming data specialization
- Bachelor's degree in Computer Science, Engineering, or related STEM field
- Expert knowledge of Agile methodologies and delivery practices
- Provide technical leadership and mentorship to junior data engineers
- Communicate complex streaming concepts to technical and non-technical stakeholders
