The Company is looking to build and manage modern data pipelines on the AWS platform to ensure scalable, efficient, and reliable data solutions.
Requirements
- At least three (3) years experience in software engineering, with demonstrated experience in CI/CD principles.
- Experience in monitoring and alerting of integrated systems, using tools such as Grafana and PagerDuty.
- Experience with Agile software development methodologies
- Experience with data orchestration and workflow management tools such as Apache Airflow and Git
- Proficiency in AWS services such as Glue, Macie, S3, Redshift, and Databricks / Snowflake.
- Strong programming skills in languages including Python, SQL. Experience with C-Sharp or Java is a plus.
- Experience with data modeling, ETL/ELT processes, and data integration techniques and tools
Responsibilities
- Collaborate with product leads and stakeholders analyze user requirements, identify functional and non-functional needs, and translate them into technical specifications.
- Interpret and contribute to data pipeline architecture, including component breakdown, data flow, and system interactions, considering scalability, performance, and maintainability.
- Write clean, efficient, and well-documented code (SQL, Python, PySpark) and frameworks, adhering to coding standards and best practices.
- Conduct thorough unit testing to ensure code quality, identify bugs, and implement necessary fixes.
- Integrate work product into the overall system, perform system testing, and validate upon deployment into UAT and PROD environments.
- Monitor data platform performance, address production issues, implement bug fixes, and provide ongoing maintenance and support.
- Create and maintain technical documentation, including design documents, data catalogs, data lineage, and data quality.
Other
- Bachelor’s degree in computer science or related field, or equivalent work experience and/or equivalent combination of education and experience
- Must be willing and able to work a flexible schedule to include holidays, nights, and weekends
- Work in a fast-paced, busy, and somewhat stressful environment
- Ability to read and interpret documents in English, such as safety rules, operating and maintenance instructions, and procedure manuals. Ability to read and communicate verbally in English. Written communication skills in English may also be required.
- Ability to apply commonsense understanding to carry out instructions furnished in written, oral, or diagram form.
