Leveraging big data technologies to gain new insights into customer experiences by building data frameworks, ingestion pipelines, and tools.
Requirements
- Proficiency in developing Software for Java (Spring & Springboot), Scala for spark streaming & spark applications, or other JVM based languages.
- Working Knowledge of SQL, XML, JSON, YML, very strong Python and Linux
- Knowledgeable with tools and frameworks Docker, Spark, Scala, Jupiter Notebook, Databricks Notebook, Kubernetes, Feature Management Platforms, SageMaker
- Advanced experience with scripting language – Python or Shell is a must have
- Experience with cloud platforms such as AWS, Azure or GCP - Amazon web services: EC2, S3, and EMR (Elastic Map Reduce) or equivalent cloud computing approaches
- Strong expertise in Data Warehousing and analytic architecture
- Experience with building stream-processing applications using Spark Streaming, Flink, etc. is a plus
Responsibilities
- Designing/developing ETL jobs across multiple big data platforms and tools including S3, EMR, Hive, Spark SQL, PYSpark.
- Clean, transform and validate data for use in analytics and reporting
- Monitor data quality and pipeline performance, troubleshoot and resolve data issues
- Participate in the entire product lifecycle for data, software products and services that are broad in scope and complexity, applying a full understanding of software engineering methodologies and industry best practices for data products.
- Rapidly fix bugs and solve problems
- Code reviews and Defect remediation
- Actively stay abreast of industry best practices, share learnings, and experiment and apply cutting edge technologies while proactively identifying opportunities to enhance software applications with AI technology
Other
- 70-85% hands-on development in all phases of the software life cycle.
- Collaborates effectively with senior engineers and architects to solve problems spanning their respective areas to deliver end-to-end quality in our technology and customer experience.
- Influences and communicates effectively
- Experience with Agile Development, SRUM, and/or Extreme Programming methodologies
- BS or MS in Computer Science, Data Engineering or related field
