Altimate AI is revolutionizing enterprise data operations through AI, aiming to alleviate the burden on overworked data teams by automating and accelerating data tasks with solutions like DataPilot and DataMates.
Requirements
- Deep understanding of SQL Abstract Syntax Tree (AST) and experience working with SQL parsers (e.g., sqlglot) for generating column-level lineage and dynamic ETLs
- Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake
- Experience in building data pipelines using Airflow or dbt
- Solid understanding of cloud platforms, particularly AWS
- Familiarity with Kubernetes (K8s) for containerized deployments
- Strong proficiency in Python and SQL
- 8+ years of experience in data engineering, with a focus on building scalable data pipelines and systems
Responsibilities
- Building highly performant large scale data infrastructure that can scale to 100K+ jobs and handle PB scale data per day.
- Design and implement robust, scalable data infrastructure on AWS, utilizing Kubernetes and Airflow for efficient resource management and deployment.
- Design and develop a comprehensive SQL intelligence system encompassing query optimization, dynamic pipeline generation, and data lineage tracking.
- Leverage your expertise in SQL query profiling, AST analysis, and parsing to create a sophisticated engine focused on query performance improvements, building adaptive data pipelines, and implementing granular column-level lineage.
- Enhancing our AI capabilities
- Developing high-performance data systems
- Integrating advanced AI using the product into thousands of data teams' daily workflows.
Other
- This remote position requires an overlap with US Pacific Timezone.
- brings a passion for innovation and problem-solving, not limited to traditional data engineering.
