Packaged Agile is looking to hire a Data Engineer to architect and implement data solutions that drive business value and enable informed decision-making.
Requirements
- Minimum 7 years of experience in Software Development focused on data proficiency in designing and maintaining data architectures to meet business needs.
- Experience in acquiring, processing, and optimizing datasets.
- Ability to support AI/ML teams by enhancing feature engineering code.
- Strong familiarity with Spark, including regular updates.
- Skilled in creating, managing, and optimizing Spark Structured Streaming jobs, leveraging technologies like Delta Live Tables and/or DBT.
- Competence in developing and managing ksqlDB and Kafka Streams queries/code.
- Experience maintaining and updating Python-based data processing scripts executed on AWS Lambdas.
Responsibilities
- Collaborate with Artificial Intelligence and Machine Learning team members to operationalize data pipelines and ML tasks.
- Provide day-to-day support for deploying Python-native ML pipelines.
- Implement efficient data ingestion strategies and change data capture mechanisms, leveraging Databricks and AWS services, to ensure real-time and accurate data updates for analytics and decision-making processes.
- Utilize Databricks to design, develop, and maintain scalable data pipelines, enabling efficient extraction, transformation, and loading (ETL) processes.
- Leverage Python programming expertise to enhance data processing scripts, ensuring robustness, efficiency, and maintainability in AWS Lambda environments.
- Execute data engineering tasks to facilitate AI/ML capabilities.
- Assist in architectural leadership, technical support, and advisement services to ensure integration of identity management system technologies meets security requirements.
Other
- US Citizenship with the ability to receive a Public Trust clearance or higher
- Must be eligible to obtain a Public Trust Clearance.
- Communicate results effectively to diverse audiences through presentations.
- Provide support to leadership engaging with senior-level executives at a public facing Federal agency, offering subject matter expertise in security architecture and other key domains.
- Strongly preferred: candidates who live in an SBA HUBZone
