Observability Best Practices

Chapter 1 · Practice 1 of 4

Define service level indicators as good events over valid events

From Deciding what to measure

Why it matters

A service level indicator (SLI) is a measurement of something users care about. The Google SRE workbook recommends expressing every SLI as a ratio: good events divided by valid events, so it always sits between 0% and 100%.

For a request-driven service, the workbook's starting SLIs are:

SLIGood event
AvailabilityThe request got a successful response
LatencyThe request was faster than a threshold
QualityThe response was not degraded (for example, no fallback results)

The workbook advises five or fewer SLI types covering the most critical functionality. More than that and nobody knows which ones matter.

Measure as close to the user as you can afford. Load balancer logs see failures that your app never logged; client-side measurement sees network problems the load balancer never saw. Pick one implementation, write down what it misses, and improve it later.

Example

PROMQL
# Availability SLI over 5 minutes (PromQL)
sum(rate(http_requests_total{job="checkout", code!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="checkout"}[5m]))