Observability Best Practices

Chapter 1 · Practice 2 of 4

Set SLOs below 100% and agree what happens when the budget runs out

From Deciding what to measure

Why it matters

A service level objective (SLO) is a target for an SLI over a window, such as 99.9% of requests successful over 28 days. The gap to 100% is the error budget: the amount of failure you accept.

The SRE workbook argues that 100% is the wrong target. Dependencies fail anyway, each extra nine costs more and helps users less, and a 100% target forbids the change you need to improve the service.

An SLO only changes behaviour if there is an error budget policy, written down and agreed by product, development and operations. Typical actions when the budget is spent:

  • prioritise reliability work over features
  • pause risky launches
  • freeze non-urgent production changes until the service is back within its SLO

The workbook is clear that your first SLO does not need to be right. Start from current performance, compare SLO misses with real incidents and support tickets, and adjust.

Do

  • Write the error budget policy down and get it signed off
  • Review SLOs against real incidents every quarter

Avoid

  • A 100% target for anything
  • SLOs that nobody uses to make decisions