American Express is looking to develop new products and platforms for their customers in the digital payments space, aiming to build interactive experiences at all touch points of a consumer's journey. This involves creating new payment vehicles, partnerships, and interactive mobile experiences.
Requirements
- GCP services like Dataflow ( Apache Beam)
- Cloud Composer ( Apache Airflow)
- Experience developing data ingestion processes from various sources (API’s, databases, flat files)
- Cleaning, enriching, and transforming raw data using tools like BigQuery SQL
- Write efficient, complex SQL queries for data transformation, validation and reporting.
- CI/CD, testing and documentation for data pipeline development.
- version control tools like Git, Jenkins etc.
Responsibilities
- Responsible for Design, Build and maintain scalable data pipelines using GCP services like Dataflow ( Apache Beam)
- Orchestrate workflows using Cloud Composer ( Apache Airflow) to mange pipeline scheduling and dependencies.
- Experience developing data ingestion processes from various sources (API’s, databases, flat files)
- Experience in Cleaning, enriching, and transforming raw data using tools like BigQuery SQL and document data lineage and transformations and Implementing data quality checks.
- Write efficient, complex SQL queries for data transformation, validation and reporting.
- Monitor data pipeline performance, troubleshoot issues, and ensure data quality and integrity.
- Implement best practices in CI/CD, testing and documentation for data pipeline development.
Other
- Proven technical capabilities and ability to work collaboratively and optimally with a range of stakeholders
- Worked in a product company
- Knowledge of developing trends and emerging standards in payments domain
- Attention to detail and strong organization skills
- Flexible working model with hybrid, onsite or virtual arrangements depending on role and business need
