arrivia is seeking a Senior Data Engineer III to manage, transform, and analyze large volumes of data using Microsoft SQL Server, SSIS, mySQL, and Azure, with a focus on creating datasets for business intelligence and e-commerce site search, and leading the adoption of AI/ML for new insights and efficiencies.
Requirements
- Proficient in SQL and relational database concepts, with a solid understanding of advanced ETL/ELT processes and data transformation.
- Experienced in working with structured and semi-structured data formats, including JSON, XML, CSV, and TXT, as well as managing data pipelines using SSIS (SQL Server Integration Services) or equivalent tools.
- Experience with C-Sharp programming.
- Experience with DTS packages, Azure services, Redis cache and message queues
- Hands-on experience or familiarity with Azure Data Factory, Databricks, or Spark is preferred.
- Strong knowledge of batch data processing and executing batch transactions for DML operations in production environments.
- Skilled in query optimization, with experience interpreting and leveraging execution plans to improve database performance.
Responsibilities
- Design, develop, and maintain data pipelines to import, clean, and transform data from multiple structured and semi-structured sources, including JSON, XML, and CSV files.
- Aggregate and create optimized datasets to support business intelligence reporting, analytics, and e-commerce site search functionality.
- Monitor and troubleshoot data pipelines, proactively addressing performance bottlenecks, data quality issues, and system inconsistencies.
- Drive the adoption of AI-assisted tools (e.g., GitHub Copilot, ChatGPT, Azure OpenAI) to enhance productivity, code quality, and innovation across the team.
- Troubleshoot and resolve data-related issues, including root cause analysis, to ensure the reliability and performance of data systems.
- Implement best practices for query optimization, leveraging execution plans and advanced SQL techniques, such as window functions, to ensure efficient data handling.
- Lead the evaluation and integration of cloud and distributed data technologies such as Azure, Databricks, and Spark.
Other
- 6 years plus of hands-on experience in data engineering, data management, or similar roles, with demonstrated expertise in working with large datasets and data pipelines.
- Bachelor’s degree in Computer Science, Information Technology, or a related field, or equivalent relevant experience in data engineering or related disciplines.
- Comfortable working independently to solve complex data engineering challenges while collaborating effectively within cross-functional teams.
- Experience thriving in a high-paced Agile development environment
- Proactive learner with a passion for leveraging modern data engineering tools and techniques to drive continuous improvement.
