Incident Response Checklist

Chapter 1 of 7 · 7 checks · 4 min

Before an incident: preparation

Checks that the people, tools and written procedures exist before anything breaks, because there is no time to create them during an incident.

Show
  1. Critical alerts page a named on-call person, with a backupMust

    Why it matters and how to check

    An alert that goes to a busy chat channel or a shared inbox is an alert nobody owns. Every production service needs an on-call rotation where urgent alerts page one named person, with automatic escalation to a second person if the first does not acknowledge.

    To verify: open your paging tool and confirm every production service has a schedule with no gaps over the next month, including weekends and holidays. Trigger a test alert for one service and time how long it takes to be acknowledged. Then leave a test page unacknowledged and confirm it escalates.

  2. Severity levels are written down with concrete, measurable examplesMust

    Why it matters and how to check

    Severity decides who gets woken up, who is told and how fast. PagerDuty's open incident response guide uses five levels, from SEV-1 (critical, warrants public notification and executive involvement) to SEV-5 (cosmetic, handled as a ticket), and suggests making definitions specific and metric driven, usually by the share of users or accounts affected. A data exposure or security vulnerability affecting customer data is a SEV-1 example in their guide.

    To verify: find your severity definitions and check each level has two or three concrete examples from your own product, a response expectation (who is paged, who is told) and a measurable trigger. Ask two engineers to classify the same past incident independently: they should land on the same level.

    Example

    Code
    SEV-1  Critical: most users cannot use the product, or customer data exposed.
       Page incident commander, tell leadership, public status update.
SEV-2  Major: a core feature is down or badly degraded for many users.
       Page incident commander, internal stakeholders informed.
SEV-3  Minor: partial loss of function, or likely to become SEV-2.
       High-urgency page to the owning team.
SEV-4  Low: no customer impact yet (one host down, a failed batch job).
       Low-urgency page, fix during working hours.
SEV-5  Cosmetic: ticket for the owning team.
  3. Runbooks exist for the most likely failures and link from the alertsShould

    Why it matters and how to check

    At 3 a.m., a responder should not have to rediscover how to fail over the database or roll back a deploy. Google's SRE book lists preparation (documenting procedures in advance, with input from the people who will use them) among its incident management best practices.

    To verify: list your top alerts by frequency and impact. Each should link to a runbook that says what the alert means, how to confirm it, the first safe actions (rollback, failover, scaling, disabling a feature flag) with exact commands, and who to escalate to. Ask someone who did not write a runbook to follow it in staging and note where they got stuck.

  4. Incident roles are defined and several people can fill each oneShould

    Why it matters and how to check

    Google's SRE incident model separates the incident commander (holds the overall picture and assigns work), operations (the only people changing the system), communication (updates for everyone else) and planning (longer-term tasks, handoffs, tracking temporary changes). PagerDuty's guide adds a scribe. Knowing these roles in advance avoids five people debugging while nobody coordinates.

    To verify: check that your incident process document names the roles and what each does. List people trained as incident commander; there should be enough to cover every on-call shift. Google suggests rotating roles so everyone becomes familiar with each one.

  5. Responders already have the access they need, including break-glass accessShould

    Why it matters and how to check

    Waiting for an access request to be approved while production is down is a common delay. On-call engineers need read access to logs, metrics and dashboards, and a documented way to get elevated access for emergencies that is logged and reviewed afterwards.

    To verify: ask this week's on-call person to open each dashboard, log search and console they would need for their services, from the device they would use at night. Test the break-glass procedure in staging: it should work in minutes, notify the security team, and leave an audit trail.

  6. Contact lists and communication channels work even if your main systems are downShould

    Why it matters and how to check

    If your chat tool, email or status page runs on the same infrastructure that failed, or if an attacker may be reading them, you need an alternative. Keep a contact list for engineering leads, security, legal, support, communications and critical vendors (cloud provider support plans, payment provider, DNS registrar).

    To verify: check the contact list is current and stored somewhere reachable during an outage, such as a printed or offline copy. Confirm the status page is hosted independently of your production stack. Agree in advance on a backup channel, such as a phone bridge, for when chat is down or possibly compromised.

  7. The team practises with tabletop exercises or game daysShould

    Why it matters and how to check

    A process used only during real incidents stays unfamiliar. Google's SRE book recommends using the incident process routinely so it becomes second nature. A tabletop exercise walks through a scenario on paper; a game day injects a real failure in staging or, carefully, in production.

    To verify: check when the last exercise happened and what it changed. Run one per quarter at least, alternating operational scenarios (database failover, bad deploy) with security ones (leaked cloud key, compromised laptop). Each exercise should produce at least one fix to a runbook, alert or contact list.