Crusoe is looking to build and operate an observability platform at scale to support their High Performance Compute (HPC) Clusters with GPUs, enabling engineers to understand the internal state of distributed systems through metrics, logs, and traces.
Requirements
- Deep expertise with metrics systems (Prometheus, Thanos, Mimir, Cortex), logging pipelines (Fluent Bit, Vector, Loki, ELK/Opensearch), and tracing platforms (Jaeger, Tempo, OpenTelemetry)
- Strong programming skills in Go or Python for automation, operators, and custom integrations
- Experience running observability platforms on Kubernetes and operating them at scale across multi-datacenter environments
- Proven ability to design, optimize, and scale telemetry pipelines handling high cardinality and high throughput data
- Solid understanding of distributed systems, performance engineering, and debugging complex workloads
- Familiarity with service meshes, networking, and workload instrumentation (Envoy, Istio, OpenTelemetry SDKs)
Responsibilities
- Designing and operating scalable observability systems (metrics, logging, tracing) across multi-datacenter Kubernetes environments
- Architecting end-to-end telemetry pipelines, including ingestion, storage, querying, and visualization
- Extending monitoring and alerting with Prometheus, Alertmanager, Thanos/Cortex, Grafana, and OpenTelemetry
- Building scalable log collection and processing pipelines with Fluent Bit, Vector, Loki, or ELK/Opensearch stacks
- Implementing distributed tracing platforms (Tempo, Jaeger, OpenTelemetry) and integrating with service meshes, load balancers, and APIs
- Automating provisioning and scaling of observability infrastructure with Kubernetes, Terraform, and custom tooling (Go, Python)
- Ensuring reliability and cost efficiency of telemetry pipelines while supporting high-volume workloads (AI/ML, HPC clusters, GPU infrastructure)
Other
- 6+ years of experience with distributed systems, with a focus on observability and monitoring systems
- Defining and driving adoption of SLOs, SLIs, and error budgets across services and teams
- Embedding security best practices into observability platforms, including RBAC, TLS, secret management, and multi-tenant access controls
- Mentoring engineers and shaping Crusoe’s observability strategy and technical roadmap
- Strong collaboration skills and the ability to influence engineering teams to adopt observability best practices
