The Company is looking to build modern data pipelines and requires a Data Software Engineer to develop, implement, and manage data solutions on the AWS platform, focusing on scalable data pipelines, optimized workflows, and data infrastructure integrity.
Requirements
- demonstrated experience in CI/CD principles.
- Experience in monitoring and alerting of integrated systems, using tools such as Grafana and PagerDuty.
- Experience with data orchestration and workflow management tools such as Apache Airflow and Git
- Proficiency in AWS services such as Glue, Macie, S3, Redshift, and Databricks / Snowflake.
- Strong programming skills in languages including Python, SQL. Experience with C-Sharp or Java is a plus.
- Experience with data modeling, ETL/ELT processes, and data integration techniques and tools
- Knowledge of data governance, data quality, and data security principles
Responsibilities
- Develop, implement, and manage the Company’s data solutions on the Amazon Web Services (AWS) platform.
- Designing and building scalable data pipelines, optimizing data workflows, and ensuring the availability and integrity of our data infrastructure.
- Interpret and contribute to data pipeline architecture, including component breakdown, data flow, and system interactions, considering scalability, performance, and maintainability.
- Write clean, efficient, and well-documented code (SQL, Python, PySpark) and frameworks, adhering to coding standards and best practices.
- Conduct thorough unit testing to ensure code quality, identify bugs, and implement necessary fixes.
- Integrate work product into the overall system, perform system testing, and validate upon deployment into UAT and PROD environments.
- Monitor data platform performance, address production issues, implement bug fixes, and provide ongoing maintenance and support.
Other
- Collaborate with product leads and stakeholders analyze user requirements, identify functional and non-functional needs, and translate them into technical specifications.
- Work effectively within cross-functional teams, product leads, and other engineers to deliver work on time and with appropriate scope & quality.
- Stay current on emerging technologies, explore new development tools and methodologies to optimize software development processes.
- Continuously identify and implement opportunities for automation, optimization, and streamlining of data engineering processes
- Must be willing and able to work a flexible schedule to include holidays, nights, and weekends
