Observability Best Practices

Chapter 1 · Practice 3 of 4

Cover the four golden signals for every service

From Deciding what to measure

Why it matters

The SRE book's chapter on monitoring distributed systems says that if you can only measure four things about a user-facing system, measure these:

SignalWhat it is
LatencyTime to serve a request. Track successful and failed requests separately, because fast errors make averages look better
TrafficDemand on the system, such as requests per second
ErrorsRate of failed requests, explicit (HTTP 500) and implicit (a 200 with the wrong content)
SaturationHow full the service is, for the resource that runs out first. Systems often degrade before 100%

These four give every dashboard the same first row, so anyone on call can read any service's health in seconds.

To check, open the main dashboard for each service and confirm all four are on it, with errors and latency split by endpoint or operation.

Avoid

  • Dashboards that start with CPU and memory instead of user-facing signals