DDN's A3I solutions are transforming the landscape of AI infrastructure. DDN is the global leader in AI and multi-cloud data management at scale. Our cutting-edge data intelligence platform is designed to accelerate AI workloads, enabling organizations to extract maximum value from their data. The Sr. Staff Software Engineer will join the Infinia Data Engine team, responsible for powering high-performance, AI-native data workflows on DDN's next-generation distributed data platform.
Requirements
- 5+ years in distributed systems, data platforms, or big data technologies.
- Expert-level knowledge of SQL, Python, and Java or Scala.
- Experience working with Apache Spark, distributed query engines, or distributed databases.
- Strong familiarity with HDFS, Hive Metastore, and data partitioning strategies.
- Hands-on experience with Apache Iceberg and/or Delta Lake.
- Deep understanding of file formats including Parquet, ORC, Avro, and their performance characteristics.
- Prior experience with C++.
Responsibilities
- Design autonomous logic for optimizing SQL and non-SQL analytic queries to leverage Infinia’s distributed infrastructure.
- Implement high-performance indexing for structured and non-structured data, using strategies such as B-epsilon trees, full-text indexing, and vectorization.
- Develop internal systems for high-throughput data access and transformation using formats such as Parquet, ORC, and Avro.
- Engineer integration layers that support open interfaces like Trino, Apache Spark, Apache Iceberg, Delta Lake, HDFS, and Hive Metastore, enabling seamless compatibility with open-source clients.
- Build and tune execution plans that leverage Infinia’s high-throughput I/O and compute capabilities for large-scale AI and analytics workloads.
- Analyze and optimize performance of distributed query execution, data storage, caching, and memory usage.
- Contribute to relevant open-source ecosystems, where appropriate, through collaboration, feature integration, or direct code contributions.
Other
- Provide technical leadership, mentorship, and design direction to other engineers on the team.
- Partner with Data Scientists, Platform Engineers, and Product Managers to deliver integrated, end-to-end solutions.
- This position requires participation in an on-call rotation to provide after-hours support as needed.
- Strong prioritization skills essential.
- Strong communication skills in all our engineers and researchers.
