Chapter 2 of 7 · 7 checks · 3 min

Reliability and capacity

Whether the service meets an agreed target under real load, and fails in contained, predictable ways when something it depends on goes wrong.

Show

SLOs are defined, measured and agreed with stakeholdersMust
Why it matters and how to check
Without a service level objective, nobody can say whether the service is healthy enough, and every incident turns into an argument. The SRE workbook recommends SLIs expressed as good events over valid events, such as the share of requests that succeed and the share faster than a latency threshold, and five or fewer SLI types per service.
To verify, find the written SLOs for the service, the query that measures each one, and the current values on a dashboard. Confirm product and engineering agreed them, along with what happens when the error budget runs out.
The service has been load tested beyond expected peakMust
Why it matters and how to check
The SRE launch checklist asks for load and end-to-end tests to find capacity at the maximum acceptable latency, and for estimates of launch-day spikes and growth. Guessing capacity is how launches fall over.
To verify, find the load test results: the request rate tested, the latency at that rate and where the service broke. The tested rate should be comfortably above the expected peak, and the result should name the bottleneck (database, CPU, a downstream API) so you know what to scale.
No single instance, node or zone failure takes the service downMust
Why it matters and how to check
The launch checklist asks how the service responds to the loss of a machine, a rack or a cluster. A single replica, or several replicas on one node or in one zone, turns routine maintenance into an outage.
To verify, check the replica count and placement: on Kubernetes, kubectl get pods -o wide should show at least two replicas on different nodes, spread across zones where the cluster has them, with a PodDisruptionBudget. Then delete one replica under load and confirm no user-visible errors.
Example
Shell
kubectl get pods -l app.kubernetes.io/name=checkout -o wide kubectl get pdb
Every outbound call has a timeout, and deadlines propagateMust
Why it matters and how to check
A call with no timeout waits forever when a dependency hangs, holding a thread, a connection or memory until the service runs out. The SRE book's chapter on cascading failures says it is usually wise to set a deadline, and recommends deadline propagation: downstream calls get the time remaining on the original request, not a fresh fixed value.
To verify, list every HTTP client, database driver and queue client in the code and check each has an explicit timeout. Then add artificial latency to one dependency in staging and confirm requests fail fast instead of piling up.
Avoid
Relying on library defaults, which are sometimes infinite
Retries use exponential backoff with jitter and are limitedMust
Why it matters and how to check
Retries that fire immediately and without limit turn a brief overload into a full outage, because every client multiplies its load at the worst moment. The SRE book says to always use randomised exponential backoff, limit retries per request, consider a server-wide retry budget and avoid retrying at several layers at once.
To verify, find the retry settings in each client: maximum attempts, backoff and jitter. Check that permanent errors such as 400 are not retried, and that only one layer of the stack retries a given call.
The service sheds load and degrades instead of falling overShould
Why it matters and how to check
Under overload, the SRE book advises failing early and cheaply, and serving a cheaper result where possible rather than failing outright. It also warns that rarely used degraded paths tend not to work, so they need testing.
To verify, identify what the service does when an optional dependency is down (recommendations, avatars, analytics) and confirm it still serves the core request. Run a load test past capacity and confirm the service rejects excess requests quickly, for example with 429 or 503, and recovers when load drops.
Readiness and liveness checks reflect real healthShould
Why it matters and how to check
The platform uses health checks to route traffic and restart stuck processes. Kubernetes warns that liveness probes must only indicate unrecoverable failure; one that checks a shared database restarts every replica when the database blips.
To verify, read the probe configuration. Readiness should fail when the instance cannot serve; liveness should check only the process itself, or be absent. Stop the database in staging and confirm Pods go unready but are not restarted.