The Department of Administration's Division of Enterprise Applications is seeking a Data Engineer to join their team to maintain the SC Enterprise Information System (SCEIS). SCEIS is the backbone of their business processes and empowers decision makers to improve government for the citizens. The role focuses on developing robust data pipelines, interfaces for data access, automating data workflows, and ensuring high data quality and performance across the organization's data platform.
Requirements
- Proven experience with data lakehouse architectures.
- Expertise in designing and optimizing ELTETL workflows in distributed computing environments.
- Proficient in writing modular, reusable, and testable code for data ingestion and transformation.
- Python and SQL.
- Hands-on experience with Apache Spark.
- Understanding of batch and real-time data processing paradigms.
- Strong grasp of data modeling concepts (dimensional, normalized/denormalized, star/snowflake schemas) and data warehousing tools and concepts.
Responsibilities
- Design, build, and test ELTETL pipelines responsible for data ingress, curating, and warehousing of data based on functional and non-functional requirements.
- Develop and maintain enterprise data cataloging processes to ensure datasets are properly documented, discoverable, and accessible, including managing metadata, data lineage, classifications, and integration with data governance tools.
- Develop and manage the deployment of data pipelines and applications through CICD pipelines andor GitOps based automation.
- Work with architecture, productservice owners, and the development team to clarify work item functional and non-functional requirements including tasks needed to complete the work item so accurate effort estimates can be provided in backlog refinement meetings.
- Evaluate new and emerging technologies to determine which ones can help innovate andor implement new feature requests.
- Participate in peer code reviews to ensure code quality and consistent use of patterns and styles.
Other
- A bachelor's degree in computer science, information technology, or a related technical field and 2 years of experience in cloud-based data platforms (e.g. Azure Synapse, AWS Redshift, Google BigQuery, Databricks, Snowflake, etc.
- Relevant experience may substitute for the bachelor’s degree on a year-for-year basis.
- Familiarity with Java is a plus.
- Experience working in agile development environments.
- Applicants indicating college credit or degree(s) on the application will be required to bring a copy of college transcript to the interview.
