The ETL Developer position is responsible for the establishment and implementation of new or revised application systems and programs, contributing to applications systems analysis and programming activities. The role aims to process, transform, and drive insights from large-scale data using distributed data processing jobs.
Requirements
- Experience in systems analysis and programming of software applications
- Design and Implement of Spark applications to process and transform large datasets in HDFS.
- Develop ETL Pipelines in Spark using Python for data Ingestion, cleaning, aggregation, and transformations.
- Optimize Spark jobs for efficiency, reducing run time and resource usage.
- Finetune memory management, caching, and partitioning strategies for Optimal performance
- Load data from different sources into HDFS, ensuring data accuracy and integrity.
- Integrate Spark Applications with Hadoop frameworks like Hive, Sqoop etc.
Responsibilities
- Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
- Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
- Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
- Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
- Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
- Design and Implement of Spark applications to process and transform large datasets in HDFS.
- Develop ETL Pipelines in Spark using Python for data Ingestion, cleaning, aggregation, and transformations.
Other
- 5-8 years of relevant experience
- Experience in managing and implementing successful projects
- Working knowledge of consulting/project management techniques/methods
- Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
- Has the ability to operate with a limited level of direct supervision.
