NISC is seeking an experienced Data Engineer to curate and optimize their data and data pipeline architecture, as well as optimize data flow and collection for various application teams. The goal is to support application experts, software developers, database architects, and data analysts on a Data Roadmap strategy and ensure optimal data delivery architecture is consistent throughout ongoing projects.
Requirements
- Experience with AWS: Lambda, S3, SQS, SNS, CloudWatch, etc.
- Experience with Databricks and Delta Lake.
- Experience with big data tools: Hadoop, Spark, Kafka, etc.
- Experience with relational SQL and NoSQL databases, including Oracle, Postgres Cassandra, and DynamoDb.
- Experience with data pipeline and workflow management tools: Hevo Data, Airflow, etc.
- Experience with AWS cloud services: EC2, Databricks, EMR
- Experience with stream-processing systems: Apache Spark, Kafka Streams, Spring Cloud, etc.
Responsibilities
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Design and build optimal data pipelines from a wide variety of data sources using AWS technologies.
- Create data tools for analytics and data scientist team members that assist them in building and optimizing a unified data stream.
- Work with other data engineering experts to strive for greater functionality while making data more discoverable, addressable, trustworthy, and secure.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Experience building and optimizing data pipelines, architectures, and data sets.
- Build ETL processes supporting data transformation, data structures, metadata, dependency, and workload management.
Other
- Hybrid Schedule: Minimum of working 3 day per week out of an office location and ability to work up to all 5 days a week from an office location.
- Candidates working from a remote location within approved states will be considered for those who have applicable industry experience with Databricks.
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Create and maintain a culture of engagement and one that is conducive of NISC's Statement of Shared Values.
- Commitment to NISC's Statement of Shared Values.
