American Express is looking for a Senior Engineer to design, implement, and optimize Bigdata, GCP Cloud based ETL Pipelines and Spark applications to support the organization’s data-driven initiatives.
Requirements
- 8+ years of hands-on software development experience with Big Data & Analytics solutions
- Hadoop Hive, Spark, Python, shell scripting, GCP Cloud - Big Query, Airflow, DataProc, PubSub
- Strong experience in designing, developing, and optimizing data pipelines for large-scale data processing, transformation, and analysis using Big Data and GCP technologies.
- Proficiency in SQL and database systems, with experience in designing and optimizing data models for performance and scalability.
- Experience of building event processing pipelines with Kafka or GCP PubSub.
- Knowledge of distributed (multi-tiered) systems, algorithms & relational databases.
- Strong Object-Oriented Programming skills and design patterns.
Responsibilities
- Develop modern, high quality, and robust operational engineering capabilities.
- Develop software in technology stack which is constantly evolving but currently includes Big data platforms like BigQuery, Airflow, DataProc.
- Work with Business partners and stakeholders to understand functional requirements, architecture dependencies, and business capability roadmaps.
- Create technical solution designs to meet business requirements.
- Define best practices to be followed by team.
- Taking your place as a core member of an Agile team driving the latest development practices
- Identify and drive re-engineering opportunities, and opportunities for adopting new technologies and methods.
Other
- BS or MS degree in computer science, computer engineering, or other technical discipline, or equivalent work experience.
- Communicates effectively with product and cross functional team.
- Willingness to learn new technologies and leverage them to their optimal potential.
- Understanding of various SDLC methodologies, familiarity with Agile & scrum ceremonies.
- Employment eligibility to work with American Express in the U.S. is required as the company will not pursue visa sponsorship for these positions
