Tauseef FayyazChapter 02
Observability: metrics, logs and traces
Going past a first dashboard: which signals to collect, how Prometheus stores them, PromQL you use every week, cardinality, the OpenTelemetry Collector and sampling.
Four golden signals
Also called latency, traffic, errors, saturation, golden signals
- What
- Latency, traffic, errors and saturation: the four measurements Google's SRE book says to watch first on any user-facing system.
- Why
- They cover what users feel (slow, failing) and what predicts trouble (load, fullness), so one dashboard per service answers most first questions in an incident.
Watch out
Mixing the latency of failed requests into the average: fast errors make a broken service look quick.
RED and USE methods
Also called RED method, USE method, rate errors duration, utilisation saturation errors
- What
- RED (rate, errors, duration) is a checklist for request-driven services. USE (utilisation, saturation, errors) is a checklist for resources such as CPU, disks, queues and connection pools.
- Why
- Using RED for services and USE for the resources beneath them gives a consistent set of panels, so anyone on call knows where to look.
Watch out
Applying only USE to a service: CPU at 40 per cent says nothing about whether requests are failing.
Counter, gauge and histogram
Also called counter, gauge, histogram, summary, metric types
- What
- A counter only goes up (requests served). A gauge goes up and down (queue length). A histogram counts observations into buckets (request durations), so percentiles can be computed later across many instances.
- Why
- The type decides which functions make sense: rate on counters, raw values or averages on gauges, histogram_quantile on histograms.
Watch out
Using a summary for latency and then trying to average its percentiles across pods, which is mathematically meaningless.
PromQL basics
Also called PromQL, rate, sum by, histogram_quantile, Prometheus query
- What
- Prometheus's query language. rate() turns a counter into per-second change, sum by (label) aggregates across instances and histogram_quantile() reads a percentile from histogram buckets.
- Why
- Dashboards, SLO recording rules and alerts are all PromQL, so reading and writing it fluently is a daily SRE and platform skill.
# requests per second per service
sum by (service) (rate(http_requests_total[5m]))
# error ratio
sum(rate(http_requests_total{code=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
# p99 latency from a histogram
histogram_quantile(0.99,
sum by (le) (rate(http_request_duration_seconds_bucket[5m])))Watch out
Summing before taking rate(), or dropping the le label before histogram_quantile(), which silently gives wrong numbers.
Recording rules
Also called recording rule, precomputed query, rules file
- What
- Prometheus rules that evaluate a query on a schedule and store the result as a new, cheaper series, such as a 5 minute error ratio per service.
- Why
- SLO dashboards and burn-rate alerts query the same expensive ratios constantly; precomputing them keeps queries fast and alerts consistent.
groups:
- name: slo
rules:
- record: job:http_errors:ratio_rate5m
expr: |
sum by (job) (rate(
http_requests_total{code=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total[5m]))Watch out
Naming recording rules at random, so nobody can tell what a series means; follow the level:metric:operations convention.
Cardinality
Also called high cardinality, label explosion, series count
- What
- The number of unique time series a metric produces: every distinct combination of label values is a separate series stored in memory.
- Why
- Cardinality drives the memory, cost and speed of every metrics backend. It is the most common reason a Prometheus server falls over.
Watch out
Putting user IDs, full URLs or request IDs in labels; those belong in logs and traces, not metrics.
OpenTelemetry Collector
Also called OTel Collector, collector pipeline, receivers processors exporters
- What
- A vendor-neutral service that receives telemetry, processes it (batching, filtering, adding attributes, sampling) and exports it to one or more backends, configured as pipelines of receivers, processors and exporters.
- Why
- Applications send to one local endpoint, and the platform team can change backends, drop noisy data or add sampling without redeploying every service.
receivers:
otlp: { protocols: { grpc: {}, http: {} } }
processors:
batch: {}
exporters:
otlphttp: { endpoint: https://traces.example.internal }
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]Watch out
Defining a processor but never adding it to a pipeline, so it silently does nothing.
Head and tail sampling
Also called sampling, head-based sampling, tail-based sampling
- What
- Head sampling decides whether to keep a trace when it starts, for example keep 10 per cent. Tail sampling decides after the trace finishes, so it can keep every error and every slow trace.
- Why
- Storing every trace at scale is expensive. Sampling keeps cost down; tail sampling keeps the traces you actually open during an incident.
Watch out
Head sampling at 1 per cent and then wondering why the one failed checkout you need was never recorded.
Observability as a platform service
Also called paved observability, default dashboards, telemetry defaults
- What
- The platform ships telemetry by default: a collector on every cluster, auto-instrumentation, standard labels, and a generated dashboard and SLO template for every new service.
- Why
- Teams get useful signals on day one without becoming Prometheus experts, and labels stay consistent across the company.
Watch out
Letting each team pick its own label names, so cross-service queries and cost reports never line up.