Measure latency with histograms and percentiles, not averages

Averages hide the slow requests users complain about. The SRE book gives the example of a service with a 100 ms mean latency where 1% of requests take several seconds.

The book recommends collecting request counts bucketed by latency, which is what a histogram is, rather than averages. From a histogram you can compute percentiles such as p50, p95 and p99, and you can count "requests faster than the SLO threshold" directly for a latency SLI.

Practical points:

• Choose bucket boundaries around your SLO threshold, so the SLI is exact.
• Do not average percentiles across instances; aggregate the histogram buckets and compute the percentile from the result.
• Watch p99 as well as p50: the tail is where timeouts and retries start.

# p99 latency across all instances
histogram_quantile(
  0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))
)