Agile Defense is looking for a Data Engineer to service existing products with enhancements, bug fixes and updates while also creating new analytics products at the request of the PO. The role operates in an Agile environment that continuously deploys code and requires a solid understanding of the full software lifecycle, including DevSecOps, Software Development, System Integration, Infrastructure Management, and Cyber Security related to public-facing government products.
Requirements
- Databricks notebooks, job schedules, SQL query tools
- Managing multi-hope data processing pipelines (185 separate nodes in data pipeline)
- Scripting in SQL, pyspark, and bash shell
- Postgres database management
- FlaskAPI
- AWS Cloudflare, S3, EC2, EKS, ALB
- Docker containerization
Responsibilities
- Design, build, and maintain efficient ETL/ELT data pipelines
- Develop and optimize SQL queries for large-scale data processing
- Implement data quality checks and monitoring systems
- Create and maintain database schemas and data models
- Collaborate with cross-functional teams to integrate data from multiple sources
- Optimize database performance through indexing, query analysis, and partitioning
- Automate data workflows using CI/CD pipelines
Other
- Public Trust
- Strong oral and written communication skills.
- Able to communicate effectively with Product Owners (POs) and stakeholders to understand requirements/scope and be able to translate that into user stories.
- Able to effectively communicate and coordinate between stakeholders, POs, and team members including designers, business analysts, developers, and Scrum Masters across other development teams.
- Work to become a subject matter expert for the products that are supported.
