Connsci is seeking a skilled Data Engineer to support the building of an anomaly detection platform for a large government customer, integrating financial and procurement systems, and ensuring data ingestion, transformation, and storage processes are reliable, secure, and compliant with state and federal standards.
Requirements
- At least 2 years of experience with Azure cloud services to include experience with Data Factory, Databricks, ADLS, Synapse/SQL, or similar
- At least 2 years of experience with data modeling and transformation using SQL and PySpark
- Familiarity with data governance, compliance, and security frameworks (HIPAA, PII, FERPA, CJIS)
- Experience with Azure Data Factory, API Management, and SFTP
- Experience with ETL/ELT workflows in Azure Databricks
- Experience with data quality checks (e.g., Great Expectations)
- Experience with Azure Data Lake Storage Gen2 (bronze/silver/gold architecture)
Responsibilities
- Design and maintain secure data ingestion pipelines using Azure Data Factory, API Management, and SFTP
- Develop ETL/ELT workflows in Azure Databricks to cleanse, transform, and normalize data across multiple systems
- Implement data quality checks (e.g., Great Expectations) to validate completeness, accuracy, and timeliness.
- Manage data storage in Azure Data Lake Storage Gen2 (bronze/silver/gold architecture) with evidentiary controls (immutability, hashing, lineage).
- Collaborate with Sr. Data Scientists to operationalize anomaly detection rules and risk scoring models.
- Optimize pipelines for performance, scalability, and cost efficiency across DEV, TEST, POC, PROD environments.
- Apply security best practices in alignment with NIST SP 800-53/171
Other
- Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).
- At least 5 years of experience in data engineering or ETL development
- Ability to work remotely, with preference given to candidates in the DC Metro or Florida region
- Collaboration and innovation in a team environment
- Professional growth and development opportunities
