Filtr uses AI to scan 1000+ jobs and finds postings that perfectly matches your resume

Microsoft AI (MS AI) is seeking an experienced Principal Data Platform Architect to help architect, design and implement a highly scalable and optimized data platform for processing upwards of 50TB/day of data from AI systems.

Requirements

• 5+ years of hands experience in building from ground up large-scale data processing platforms with more than 100 data pipelines.
• 5+ years experience building data pipelines using Scala, Python (PySpark), and SQL, with deep understanding of big data technologies including Apache Hadoop ecosystem, Spark, Kafka Stream/Flink, columnar and vector databases, semantic search, RDBMS, and key-value stores.
• 5+ years of experience designing and deploying scalable services on public cloud platforms such as Azure, AWS, and GCP.
• 2+ years of hands-on experience architecting distributed systems at scale, with comprehensive systems knowledge spanning bare-metal infrastructure, containerization, and networking
• 10+ years technical engineering experience building data platforms that handle large volumes of data in both batch and real time.
• Coding in languages including, but not limited to, Scala, Python, C, Java, Spark, & SQL
• Ability to identify, analyze, and resolve complex technical issues, ensuring optimal performance, scalability, and user experience.

Responsibilities

• Thought leader and owner of the architecture blueprint of data platform.
• Superior coding skills in SPARK using Scala, Python and Java
• Work collaboratively with other Platform, infrastructure, application engineers as well as AI Researchers to build next generation data platform services and capabilities.
• Ship high-quality, well-tested, secure, and maintainable code.
• Find a path to get things done despite roadblocks to get your work into the hands of users quickly and iteratively.
• architect, design and implement a highly scalable and optimized data platform for processing upwards of 50TB/day of data from AI systems.
• architect solutions that stand the test of time

Other

• possesses technical prowess, a methodical approach to problem-solving, proficiency in data platform and data pipeline technologies, and a mastery of templating to architect solutions that stand the test of time and who will bring an abundance of positive energy, empathy, and kindness to the team every day, in addition to being highly effective.
• Is a systems thinker with a bias for action and iteration
• Navigates ambiguity with condience and drives clarity
• Passionate about building resilient, extensible platforms
• Approaches work with curiousity, collaboration, and a focus on impact