Chapter 2 of 7 · 5 checks · 3 min
Detecting and declaring an incident
Checks that problems are noticed quickly, that someone officially starts the incident process, and that everyone can find where it is being handled.
Alerts fire on user-facing symptoms, not only on causesShould
Why it matters and how to check
Alerts on CPU or disk usage miss the failures you did not predict. Alerts on what users experience, such as error rate, latency and failed checkouts or sign-ins, catch any cause that hurts users. Google's postmortem chapter lists a monitoring failure that led to manual discovery as a reason to write a postmortem, which shows how much it matters.
To verify: for each critical user journey, check there is an alert on its success rate or latency. Review the last few incidents: how was each detected? Any found by a customer or a staff member before an alert is a gap to close.
Clear criteria say when to declare an incidentMust
Why it matters and how to check
Teams often lose time because nobody wants to "make it a big deal". Google's SRE book gives simple criteria: declare an incident if a second team is needed to fix it, if the outage is visible to customers, or if the issue is still unsolved after an hour of concentrated analysis. It also says declaring early is better than starting the process hours into a growing problem.
To verify: check your incident process states criteria like these, plus any security trigger (suspected unauthorised access, leaked credential, data exposure). Confirm anyone on the team, not only managers, is allowed to declare an incident, and that the command or button to do so is documented.
When severity is unclear, the higher level is chosen and reviewed laterMust
Why it matters and how to check
Debating severity during an incident wastes the minutes that matter most. PagerDuty's guide says to treat an incident as the higher severity when unsure, and to discuss the level in the postmortem instead of during the response. Downgrading later is cheap; under-reacting is not.
To verify: check your process document says this explicitly. Review recent incidents: were any started at a low severity and upgraded late? Note how long that took and use it as an example in training.
Each incident gets its own channel and a living incident documentMust
Why it matters and how to check
Google's SRE book calls a living incident document the incident commander's most important responsibility: a shared, editable record with the current state at the top, kept for the postmortem. A dedicated chat channel per incident keeps the discussion in one place and gives it a searchable history.
To verify: check you have an incident document template with sections for summary, impact, current status, roles, timeline, actions taken and follow-ups. Declaring an incident should create the channel and the document automatically, or the steps should be in the first lines of your process. Look at the last incident: was everything in one place?
Example
Code INCIDENT 2026-10-13-checkout-errors SEV-2 status: MITIGATING Commander: Aisha Ops: Bilal Comms: Sara Scribe: Omar Impact: ~30% of checkout attempts fail with 502 since 14:05 UTC Current hypothesis: 14:00 deploy of payments-api v412 Next update: 14:45 UTC Timeline (UTC) 14:05 alert: checkout success rate below threshold 14:09 incident declared, roles assigned 14:16 rollback of payments-api to v411 started (Bilal)An incident commander is named explicitly at the startMust
Why it matters and how to check
Without a named commander, the most senior or loudest person ends up coordinating while also debugging, and nobody keeps the overall picture. Google's model says the commander holds every role they have not delegated and can remove roadblocks for the people fixing the problem.
To verify: in the incident channel, the first messages should say who is commander, in a standard form such as "I am incident commander". Review recent incidents and check a commander was named within minutes of declaring. Everyone involved should be able to answer "who is leading this?" at any time.