Core & Main is looking to design, develop, test, and maintain data solutions and pipelines to enhance their data warehousing and analytics environments, ensuring high-quality, reliable data solutions that meet business needs.
Requirements
- 5+ years of hands-on development experience in SQL and/or Python for data warehouse management, data integration, and data lake management.
- Deep working knowledge in SQL development using T-SQL code to design, implement, and optimize complex database objects, such as tables, views, stored procedures, indexes, and functions.
- Experience working with Azure data architecture, including a solid understanding of tools for building data pipelines on cloud-based data platforms, such as Delta Lakehouse Medallion architecture and data warehousing solutions.
- Exposure to modern Spark-based data platforms like Databricks or Microsoft Fabric for data engineering tasks, including leveraging their capabilities for scalable data processing, analytics, and machine learning workflows in a cloud-based environment.
- Understanding of ELT vs ETL and how to build efficient data pipelines with modern Change Data Capture processes.
- Hands-on experience with CI/CD pipelines in Azure DevOps and understanding of Agile development methodologies.
- Familiarity with common data mapping and transformation techniques for Dynamics 365 Data Entities and Data Management Framework for the Finance and Operations modules.
Responsibilities
- Design, develop, test, and maintain data solutions and pipelines.
- Serve as a subject matter expert in data engineering, data integration, and cloud-based data platforms.
- Maintain, enhance, and provide solutions for data warehousing and analytics environments.
- Take ownership of technical deliverables and ensure high-quality, reliable data solutions that meet business needs.
- Develop, implement, and maintain data pipelines, data warehouses, and data lake solutions.
- Perform unit testing prior to moving code/configuration to the QA process.
- Build and maintain data models, data mappings, transformation rules, workflows, data extractions and imports, interfaces, and object models.
Other
- Collaborate with stakeholders to analyze data solution requirements, identify gaps, and assess feasibility.
- Take responsibility for the design, documentation, and implementation of technical solutions that meet business and functional requirements.
- Ensure data systems possess sufficient controls and meet compliance standards.
- Contribute to the development and definition of test plans and scripts for performance, regression, and user acceptance testing; support QA activities as required.
- Ensure data solutions comply with security protocols and data governance standards.
