The company is looking to design, develop, and manage a scalable, secure, and highly available data platform (PaaS) to meet growing user demands and facilitate efficient data ingestion, processing, storage, and analysis. This involves evolving the platform's architecture, optimizing cloud infrastructure, and automating deployment and operational processes.
Requirements
- Proven experience managing and optimizing cloud infrastructure (AWS, Azure).
- Demonstrable experience with Infrastructure as Code (IaC) tools (e.g., Terraform, CloudFormation).
- Solid experience with containerization technologies (e.g., Docker, Kubernetes).
- Experience with CI/CD pipelines and DevOps best practices (GitHub Actions preferred)
- Familiarity with data storage and processing technologies (e.g., Kafka, Spark, Flink, Hadoop, Snowflake, MongoDB, S3/ADLS).
- Proficiency in one or more programming languages commonly used for platform development (e.g., Python, Go, Java).
- Strong understanding of networking, security, and performance principles in a cloud environment.
Responsibilities
- Collaborate with stakeholders to gather requirements and translate them into a comprehensive software system. This includes designing microservices, persistence layers, and implementing deployment workflows using tools like Terraform, Ansible, Puppet, or Salt.
- Design, develop, and implement highly available, scalable, and secure platform services for our data PaaS using programming languages (e.g., Python, Go, Java).
- Contribute to the architectural evolution of our data platform, ensuring it meets the growing demands of our users and adheres to best practices.
- Develop and maintain APIs, automation tools, and internal services to streamline platform operations and enhance developer experience.
- Implement and improve data pipelines, storage solutions, and processing frameworks within the PaaS.
- Manage and optimize cloud infrastructure in AWS and Azure cloud services to support our data platform, focusing on cost-efficiency, performance, and reliability.
- Implement Infrastructure as Code (IaC) principles using tools like Terraform for provisioning and managing resources.
Other
- Bachelor's degree in Computer Science, Software Engineering, or related field. OR demonstrable equivalent deep understanding, experience, and capability.
- Master's or PhD degree in Computer Science (or related field) and 2+ years of experience in software engineering or 4+ years of experience in software engineering at a large scale. Experience in data system/platform development.
- Experience with monitoring and logging tools (e.g., Prometheus, Grafana, Datadog).
- Familiarity with database administration and optimization is a plus.
- Familiarity with application access control and automation
