BCMC is supporting a U.S. Government customer on a large mission critical development and sustainment program to design, integrate, build, deliver, and operate a network operations environment including introducing new cyber capabilities to address emerging threats. BCMC is seeking a Senior Data Engineer working to support the migration of customer applications, services and platforms to a Medallion Model.
Requirements
- 8+ years of software engineering experience.
- Minimum of 4 years of experience performing data engineering work in a cloud environment.
- Experience with relational, noSQL and/or file-based storage (e.g. Databricks, Elastic, Postgres, etc.)
- Experience with DevOps tools such as Jenkins, Docker, Ansible, Puppet, Chef, Splunk, etc.
- Working knowledge of Databricks, Cloud Relational Database Services and Elasticsearch
- Experience with Databricks workflows, Databricks Unity Catalog, Databricks Autoloader, and Databricks Delta Live Tables and Apache Spark.
- Basic working knowledge with Apache Niagara Files (NiFi)
Responsibilities
- design, develop and implement, end-to-end data ingestion and processing pipelines using various Cloud and Platform Technologies.
- Transform incoming bronze data (RAW) into silver (clean) and finally projections into gold tier. (Medallion model)
- Schedule, orchestrate and monitor automated/semi-automated data pipelines.
- Perform mapping and integration of data assets in support of the execution of multiple mission customers.
- Develop and maintain the data engineering team process improvement plan.
- Develop code using various programming and scripting languages to automate data ingestion, pipeline orchestration and improve data management processes.
- Ingest transparency, leveraging technologies such as AWS CloudWatch to identify places for measuring and gathering performance information on automated data pipelines
Other
- Must be a US Citizen
- Active Secret (S) clearance. Must be able to obtain a TS/SCI clearance
- Must be able to obtain DHS Suitability
- Demonstrate excellent oral and written communication skills to all levels of management and the customer.
- Ensure Data Engineering Team Standard Operating Procedures are appropriately captured and communicated across the team.
