Checklist · Cloud and DevOps
Incident Response Checklist
Opens with creditsAn incident response checklist for software teams, covering both outages and security incidents. It walks through preparation, detecting and declaring an incident, roles and coordination, mitigation, security containment and evidence, communication, and closing with a blameless postmortem, with a way to check each item before you need it.
Written by Tauseef Fayyaz7 chapters, 37 checksAbout 19 minReviewed 13 October 2026What you will learn
7 chapters · about 19 min
1Before an incident: preparationChecks that the people, tools and written procedures exist before anything breaks, because there is no time to create them during an incident.7 checks · 4 min
- 1Critical alerts page a named on-call person, with a backup
- 2Severity levels are written down with concrete, measurable examples
- 3Runbooks exist for the most likely failures and link from the alerts
- 4Incident roles are defined and several people can fill each one
- 5Responders already have the access they need, including break-glass access
- 6Contact lists and communication channels work even if your main systems are down
- 7The team practises with tabletop exercises or game days
7 in this chapterOpen this chapter2Detecting and declaring an incidentChecks that problems are noticed quickly, that someone officially starts the incident process, and that everyone can find where it is being handled.5 checks · 3 min
- 1Alerts fire on user-facing symptoms, not only on causes
- 2Clear criteria say when to declare an incident
- 3When severity is unclear, the higher level is chosen and reviewed later
- 4Each incident gets its own channel and a living incident document
- 5An incident commander is named explicitly at the start
5 in this chapterOpen this chapter- Coordinating the responseChecks that the response stays organised: clear roles, one group changing the system, a running timeline and clean handoffs.5 checks · 2 minOpens with credits
- Mitigation and recoveryChecks that the team stops the damage first, changes one thing at a time, and confirms recovery with data rather than hope.5 checks · 2 minOpens with credits
- When it is a security incidentChecks for incidents where an attacker may be involved: preserve evidence, contain access, rotate secrets and bring in the right people.6 checks · 3 minOpens with credits
- CommunicationChecks that stakeholders and customers hear accurate updates on a predictable schedule, from one voice.4 checks · 2 minOpens with credits
- Closing the incident and learning from itChecks that every significant incident ends with a blameless postmortem, follow-up actions that actually get done, and better runbooks.5 checks · 3 minOpens with credits
About this checklist
This incident response checklist is for the moment something breaks in production, and for the weeks before that moment. Incidents go badly for predictable reasons: the alert went to a channel nobody watched, three people made changes at once and nobody wrote them down, the team spent an hour arguing about severity, an engineer "cleaned up" a compromised server and destroyed the evidence, or customers learned about the outage from social media. None of these are technical problems. They are gaps in preparation and process, and a checklist closes them.
The structure follows the incident lifecycle used by NIST SP 800-61 Revision 3, published in April 2025, which places incident response inside the six functions of the NIST Cybersecurity Framework 2.0: Govern, Identify, Protect, Detect, Respond and Recover. The day-of-incident practices come from Google's Site Reliability Engineering book (incident command roles, a living incident document, clear criteria for declaring an incident, blameless postmortems) and PagerDuty's open incident response documentation (severity levels and how to use them).
The checklist covers operational incidents (outages, data corruption, a bad deploy) and security incidents (a leaked key, a compromised server). Most steps are the same; the security chapter covers what changes when an attacker may be involved, mainly preserving evidence and containing access. Each check has a severity: must means have it before your next incident, should means put it in place soon, nice means worth doing once the rest works.
Common questions
What are the phases of incident response?
The traditional lifecycle is preparation, detection and analysis, containment, eradication and recovery, and post-incident activity. NIST SP 800-61 Rev. 3 now places incident response inside the six NIST CSF 2.0 functions: Govern, Identify, Protect, Detect, Respond and Recover, so preparation and improvement are part of ongoing risk management.
What does an incident commander do?
The incident commander coordinates the response rather than fixing the problem directly. They keep the overall picture, assign roles such as operations and communication, keep the incident document current, remove blockers, and decide when the incident is resolved. In Google's model they hold any role they have not delegated.
When should we declare an incident?
Google's SRE book suggests declaring when a second team is needed, when customers can see the problem, or when it is still unsolved after an hour of focused analysis. Add security triggers such as a leaked credential or suspected unauthorised access. Declaring early costs little, while declaring late wastes the most valuable time.
What is a blameless postmortem?
It is a written review of an incident that focuses on contributing causes in systems and processes rather than on individual mistakes. It assumes everyone acted in good faith with the information they had, so people report problems openly. It records impact, timeline, causes and follow-up actions with owners.
How are security incidents different from outages?
The coordination is similar, but an attacker may be involved, so you preserve evidence before changing systems, contain access by isolating rather than deleting, rotate every credential the system could reach, limit who knows details, and may need a separate communication channel. Legal or privacy staff may need to be involved for notification duties.
How many severity levels should we have?
Enough to drive different responses, and few enough that people choose quickly. PagerDuty's public guide uses five, from SEV-1 critical to SEV-5 cosmetic. Whatever you choose, define each with measurable triggers and examples, and pick the higher level when unsure.
Sources (6)
Checked against these, last on 13 October 2026.
- SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management, NISTcsrc.nist.gov/pubs/sp/800/61/r3/final
- NIST Releases Version 2.0 of Landmark Cybersecurity Framework, NISTwww.nist.gov/news-events/news/2024/02/nist-releases-version-20-landmark-cybersecurity-framework
- Managing Incidents, Google SREsre.google/sre-book/managing-incidents/
- Postmortem Culture: Learning from Failure, Google SREsre.google/sre-book/postmortem-culture/
- Severity Levels, PagerDutyresponse.pagerduty.com/before/severity_levels/
- PagerDuty Incident Response Documentation, PagerDutyresponse.pagerduty.com/
Related on Sefism
Picked from the other libraries by shared topics.
Roadmaps
See allCheat sheets
See allBest practices
See all- Kubernetes Best PracticesRun workloads on Kubernetes that schedule well, survive restarts and node drains, and stay locked down: resources, probes, PDBs, Pod Security and RBAC.
- Terraform Best PracticesWrite Terraform that teams can review and change safely: file layout, version pinning, remote state with locking, secrets, modules, refactoring and tests.
- Secrets Management Best PracticesKeep API keys, passwords and tokens out of git, logs and browser bundles: secret managers, scanning, rotation, OIDC in CI and leak response.
Checklists
See all- Kubernetes Production ChecklistVerify a workload and its cluster before real traffic: requests and limits, probes, PDBs, spread, Pod Security, RBAC, network policies, secrets, images, autoscaling, upgrades and backups.
- Disaster Recovery ChecklistBackups and DR you can prove: RPO and RTO per system, backup coverage, off-site and immutable copies, restore tests, zone and region choices, runbooks, drills, dependencies and communication.
- Production Readiness ChecklistCheck a service before it takes real traffic: ownership, SLOs, capacity, timeouts and retries, alerts, rollout and rollback, backups, security and on-call.
Project structures
See all- Terraform project structureAn infrastructure repository for a small app on AWS: reusable modules, one folder per environment with its own remote state, pinned providers, tests and security scans, with plan on every pull request and apply from CI. For students who have finished the Terraform tutorials and want to know how a real team lays out the files. The same layout works unchanged with OpenTofu.
- Kubernetes with Helm project structureA Helm chart repository for one app: the chart that
helm creategives you, grown into a tested, versioned package with per-environment values, unit tests, install tests in a kind cluster and releases to an OCI registry. For students who can write a Deployment by hand and want to know how a team organises charts. It is about the repo layout and what each file does, not about cluster design. - Spring Boot project structureA REST API in Java with Spring Boot 4, PostgreSQL, Flyway and Testcontainers, organised by feature the way the Spring docs suggest. It grows from the Initializr scaffold to a service with JWT security, problem-detail errors, MapStruct DTOs, slice and integration tests, CI, containers and observability, and finally Modulith boundaries, contract and load tests. For students who know some Java and want to see how a real service is organised.
Career questions
See all- Do I need Docker and Kubernetes as a junior developer?Docker, yes: learn the basics, because nearly every team uses containers and you will be expected to run the project locally and read a Dockerfile. Kubernetes, mostly no: understand what it does and a few core terms, but deep Kubernetes can wait until your job or a DevOps direction needs it.
- What is platform engineering, and how is it different from DevOps?DevOps is a culture and set of practices: developers and operations sharing responsibility for shipping and running software. Platform engineering is one way to put that into practice at scale: a team builds an internal platform, a paved road of tools and templates, so product teams can ship without becoming infrastructure experts.
- Can I become an SRE or platform engineer straight after graduating?Sometimes, but it is rare, because these roles expect you to have run production systems. The more reliable path is a first job in backend, DevOps or cloud support, where you learn production, then a move into SRE or platform work within a couple of years.
Stuck on something specific?
Writing only gets you so far. If you want an answer to your situation rather than the general case, book a session and we will work through it together. Sessions are free for approved Sefism members, and a few slots open each week.
- 1:1 Mentorship
- Fresh Graduate Mentorship
- Career Guidance
Brand deals, promos and partnerships: tauseeffayyaz@gmail.com