An incident response runbook turns a stressful, ambiguous alert into a sequence of manageable decisions. This copy-and-customize guide covers triage, severity, ownership, communication, mitigation, rollback, evidence collection, recovery, and postmortem follow-up, with checkpoints your team can review monthly or quarterly.
Overview
An incident response runbook is an operational agreement, not just a list of commands. It tells the on-call engineer what to confirm first, who should be involved, how decisions are recorded, and when an incident can be closed. The best runbooks reduce cognitive load without preventing experienced responders from adapting to unfamiliar conditions.
Keep the document close to the systems it describes. Link to dashboards, alert definitions, deployment controls, service ownership records, status-page procedures, and access instructions. Avoid embedding credentials or sensitive customer data. Instead, describe the approved access path and identify the team responsible for granting access.
The workflow below assumes an incident may involve an application, infrastructure, deployment, dependency, security control, or data pipeline. Adapt the severity labels and notification rules to your organization. A Kubernetes-focused team can pair this process with a Kubernetes troubleshooting runbook, while teams reviewing delivery-related failures may also benefit from a release engineering KPI checklist.
Copy-and-customize incident header
Incident ID: INC-YYYY-MM-DD-###
Start time (UTC):
Detected by:
Incident commander:
Technical lead:
Communications lead:
Affected service or component:
Current severity:
Incident channel or bridge:
Customer impact:
Status page decision:
What to track
Track facts, decisions, and changes over time. A shared incident timeline is more reliable than memory, especially when several responders are working in parallel.
1. Alert and triage details
- Record the alert name, detection time, affected signal, and monitoring link.
- Confirm whether the alert is actionable, duplicated, stale, or caused by a test.
- Check the basic scope: affected regions, tenants, services, versions, and request paths.
- Compare current behavior with a known healthy period without assuming that correlation proves causation.
- Capture the first relevant logs, traces, deployment events, configuration changes, and infrastructure events.
2. Severity and ownership
Define severity using observable impact rather than the apparent technical complexity. A useful classification considers availability, data integrity, security exposure, customer reach, duration, and whether a safe workaround exists. Write down the reason for the selected level so that escalation is understandable.
Assign one incident commander to coordinate priorities and decision-making. A technical lead investigates and executes changes; a communications lead handles internal updates, customer messaging, and the status page when appropriate. One person may hold multiple roles during a small incident, but the roles should still be explicit.
3. Communication and decision records
Use a predictable update format: current impact, what changed since the last update, what is being investigated, the next action, and the next update time. Do not promise a resolution time unless the team has a defensible basis for doing so. Record rejected hypotheses and risky actions as well as successful ones.
4. Mitigation, rollback, and evidence
Prefer reversible actions that reduce impact while preserving evidence. Examples include disabling a feature flag, shifting traffic, pausing a rollout, scaling a constrained component, or reverting a known-bad release. Before changing systems, note the current version, configuration, and relevant metrics. If rollback is considered, confirm the rollback target, compatibility assumptions, database implications, and verification plan.
Preserve timestamps, command output, dashboard snapshots, trace examples, deployment identifiers, and relevant configuration diffs according to your organization’s retention and security requirements. Never copy secrets into the incident channel or ticket.
5. Recovery and closure
Recovery is more than a green dashboard. Confirm that error rates, latency, saturation, queue depth, data processing, and customer-facing workflows have returned to an acceptable state. Watch long enough to distinguish recovery from a temporary lull. Record the time mitigation began, the time service stabilized, and any remaining limitations.
- [ ] Alert acknowledged and scope confirmed
- [ ] Severity assigned and incident commander named
- [ ] Technical and communications roles assigned
- [ ] Incident channel and timeline created
- [ ] Customer impact assessed
- [ ] Mitigation selected and risk reviewed
- [ ] Rollback or change verified, if used
- [ ] Recovery signals checked
- [ ] Stakeholders notified of resolution or remaining risk
- [ ] Follow-up owner and due date recorded
Cadence and checkpoints
A runbook should be reviewed on a recurring schedule, not only after a major outage. Monthly reviews work well for fast-changing services; quarterly reviews may be sufficient for stable systems. Add an immediate review after a significant incident, architecture change, new deployment path, monitoring migration, or change in service ownership.
During the incident
- At acknowledgement: confirm the alert is actionable and assign the first owner.
- Within the initial triage window: establish impact, severity, scope, and the next communication time.
- After each major action: record the change, expected result, observed result, and rollback option.
- At stabilization: verify multiple recovery signals and identify residual risk.
- Before closure: confirm communications, evidence, ownership, and follow-up work.
Monthly or quarterly review
Sample recent incidents and ask whether responders could find the right dashboard, owner, access path, rollback procedure, and communication channel without improvising. Check for dead links, outdated service names, missing escalation contacts, alerts that lack runbooks, and instructions that require unsafe manual access. Test the process with a tabletop exercise rather than waiting for production impact. Your wider toolchain can be reviewed alongside the DevOps toolchain checklist so incident response remains aligned with delivery workflows.
How to interpret changes
Look for patterns in the process, not just individual failures. An increase in incidents may reflect better detection, a large release, or a genuine reliability regression. A decrease may indicate improvement, but it can also mean alerts are missing or teams are recording events inconsistently.
Compare related variables: time to acknowledge, time to assign ownership, time to mitigate, time to recover, repeat incidents, rollback frequency, alert noise, and the percentage of incidents with complete timelines. These measures are useful for finding friction, not for ranking individuals. A long mitigation time may point to unclear ownership, unsafe rollback procedures, insufficient observability, or a dependency that cannot be tested in isolation.
When the same symptom returns, link incidents and investigate common causes. Distinguish a recurrence from a similar-looking event with a different cause. Prioritize follow-up work that reduces future decision-making effort: better alerts, safer deployment controls, tested rollback paths, dependency timeouts, capacity safeguards, or clearer ownership.
When to revisit
Schedule the next runbook review now, then revisit it sooner when conditions change. Trigger an update after a production incident, a new service or region, a major dependency change, a migration to different monitoring tools, a new CI/CD path, a security event, or a change in on-call coverage.
Use this practical closeout sequence:
- Choose one recent incident and replay the first fifteen minutes using only the current runbook.
- Mark every missing link, unclear decision, unavailable dashboard, and undocumented access requirement.
- Assign one owner and one due date for each improvement; avoid creating ownerless action lists.
- Test the highest-risk change in a tabletop exercise or controlled environment.
- Update the runbook, version the change, and announce what responders should do differently.
- Set a monthly or quarterly calendar reminder and retain the previous revision for audit and learning purposes.
A runbook earns its place when an on-call engineer can use it under pressure and a new responder can understand the team’s operating model. Keep it concise at the point of action, specific about ownership, careful with evidence, and honest about uncertainty. Revisit it regularly so the documented process continues to match the systems it protects.
