Norfolk Southern is looking to build cutting-edge Big Data solutions and Big Data pipelines to advance the skills and capabilities of the organization, while improving the level of enablement for partners and stakeholders.
Requirements
- Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python.
- Advanced knowledge of Big Data, Data Engineering, and BI with hands-on and technical experience in tools/technologies like Apache Kafka, Spark, Hadoop, Spotfire, Tableau, PowerBI, etc.
- 5 years of experience in building data ingestion pipelines using: python/PySpark and/or Spark-Streaming, etc.
- Experience with relational databases e.g. Teradata, Oracle, SQL Server, as well as NoSQL databases including HBase and Cassandra.
- Hands-on coding experience in at least one modern programming language: Python, Scala, etc. Strong SQL and Python skills, including knowledge of Python libraries/frameworks.
- Knowledge of Unix/Linux platform and shell scripting.
- Experience with data engineering productivity toolchains and concepts (CI/CD, DevOps, GitHub, Jira, Confluence, Agile Scrum / Kanban, etc.)
Responsibilities
- Develop the vision and manage the execution of the Data Engineering roadmap by collaborating with cross-functional teams, products, analytics, engineering, and machine learning experts.
- Oversee various parts of the data lifecycle from acquirement to insights, in particular: data ingestion, data warehouse, Data Lakehouse, and data engineering.
- Lead Data Engineering team to use state-of-the-art Big Data tools and technologies to build scalable data architecture and performant data pipelines to acquire high-velocity real-time and batch data of different sizes and scales.
- Design, implement, and support scalable data infrastructure solutions to integrate with multi-heterogeneous data sources, aggregate and retrieve Big Data in a fast and safe mode, curate data that can be used in BI reporting, analysis, machine learning models and ad-hoc data requests.
- Ensure appropriate levels of governance and testing procedures for developed solutions.
- Stay up to date with technology, trends, and tools in Big Data.
- Take part in development of data pipelines and solutions.
Other
- Bachelor’s Degree, preferably in Information Systems, Computer Science, Computer Information Systems or related field.
- Experience as a People Leader, directly managing technical teams, including hiring, developing, motivating, and directing people as they work.
- Strong project management skills, including work prioritization, planning, and task delegation.
- Ability to effectively communicate (both written and verbally) technical information at all levels of the organization.
- Location: Atlanta, GA, Environment: Hybrid (three days in office; two days remote)
