Platform engineering and SRE

Stage 2 of 10 · 7 lessons · ~6 weeks

Observability: metrics, logs and traces

Go past a first dashboard: choose the right signals, write PromQL fluently, keep cardinality under control, run the OpenTelemetry Collector and connect metrics, traces and logs so you can find a cause quickly.

Start: Golden signals, RED and USE

Lessons· one lesson for each topic, in this order

  1. 1Golden signals, RED and USE
  2. 2The Prometheus data model
  3. 3PromQL you use every week
  4. 4Cardinality and what belongs in metrics
  5. 5OpenTelemetry and the Collector
  6. 6Sampling and correlating signals
  7. 7Observability as a platform default

Do this

  • Run a local cluster with kind and install Prometheus and Grafana.
  • Instrument a small app with OpenTelemetry and send traces through a Collector.
  • Write PromQL for request rate, error ratio and p99 latency, then turn two into recording rules.
  • Build one RED dashboard for the app and one USE dashboard for its database or queue.
  • Check series counts per metric and remove one high-cardinality label.
  • Add trace IDs to logs and follow one slow request from graph to trace to log.

Start with· follow one of these

Practise

Keep handy

Also worth your time

Before moving on: A demo app on a local cluster with Prometheus and an OpenTelemetry Collector, a RED dashboard built on recording rules, traces with tail sampling that keeps errors, and log lines that carry trace IDs so you can jump from a graph to a trace to a log.

Tauseef's advice
Learn the open source stack first: Prometheus, Grafana and OpenTelemetry. Paid tools use the same ideas with nicer screens, and you will understand their bills far better. Build everything on a kind cluster on your laptop so you can break it freely.

Mark this stage as not done?