SRE Incident Response Runbook Template: Triage, Communication, Mitigation, and Review
sreincident-responseon-callobservabilityreliability

SRE Incident Response Runbook Template: Triage, Communication, Mitigation, and Review

CControl Center Editorial Team
2026-11-13
7 min read

A practical SRE incident response runbook template covering triage, roles, communication, mitigation, handoffs, and blameless review.

A production incident runbook gives an on-call team a shared sequence for reducing impact, coordinating decisions, and preserving useful evidence. Use the checklist below as a starting point, then replace the placeholders with your actual services, owners, escalation paths, dashboards, and communication channels.

Overview

Incident response is easier when responders do not have to design the process during an outage. A good SRE incident management runbook answers five questions in order: what is happening, how severe is it, who is coordinating, what action is safest now, and how will the team know that the incident is over?

The runbook should support judgment rather than replace it. It should identify the first checks, provide links to trusted telemetry, define escalation conditions, and make handoffs unambiguous. It should not encourage risky changes simply because a checklist contains them. If the situation differs from the documented assumptions, record the difference and escalate to the service owner or incident commander.

Before adopting this production incident checklist, define the following placeholders:

  • Service: the affected application, dependency, or platform component.
  • Primary owner: the team accountable for operation and remediation.
  • On-call contact: the current responder and backup contact.
  • Incident channel: the chat, conference, ticket, or bridge used for coordination.
  • Telemetry: dashboards, logs, traces, alerts, deployment history, and status indicators.
  • Change controls: the approved method for rollback, configuration changes, feature flags, or traffic management.

For services deployed on Kubernetes, link the runbook to the team’s deployment strategy and operational commands rather than copying commands that may become outdated. The Kubernetes deployment strategy guide can help teams document the rollback and traffic-shifting approach appropriate to each service.

Checklist by scenario

1. First five minutes: acknowledge and establish control

  1. Acknowledge the alert or report and record the start time.
  2. Open or locate the incident record and communication channel.
  3. State the initial symptom in plain language, such as “checkout requests are returning errors for some users.” Avoid guessing at the cause.
  4. Assign or confirm the incident commander. If no formal role exists, the first responder should name an interim coordinator.
  5. Check whether the alert is a duplicate, a known maintenance event, or a symptom of a broader incident.
  6. Capture the first relevant dashboard view, alert payload, log query, and deployment or configuration change.

The initial objective is not to find the root cause. It is to establish a reliable shared picture and prevent several responders from taking conflicting actions.

2. Triage and severity assessment

Use criteria that reflect user impact and operational risk, not the loudness of an alert. A practical severity model might distinguish between:

  • Critical: a major user-facing function is unavailable, data integrity may be at risk, or the incident requires coordinated leadership and customer communication.
  • High: a significant function is degraded, error rates or latency are materially elevated, or the issue is spreading.
  • Moderate: impact is limited, a workaround exists, or the affected scope is contained.
  • Low: there is no immediate material user impact, but investigation or corrective work is still required.

For each alert, ask:

  • Which users, regions, tenants, endpoints, or workflows are affected?
  • When did the impact begin, and is it increasing, stable, or recovering?
  • Are availability, latency, correctness, privacy, or security affected?
  • Did a deployment, infrastructure change, dependency event, certificate change, or traffic shift precede the symptoms?
  • Is there a safe mitigation that reduces impact without destroying evidence?
  • What condition would require escalation to another team or a higher severity?

Record uncertainty explicitly. “Cause unknown; impact appears limited to API writes in one region” is more useful than an unsupported diagnosis.

3. Roles, escalation, and communication

At minimum, assign these responsibilities when the incident warrants more than one responder:

  • Incident commander: sets priorities, approves major actions, and maintains the operational timeline.
  • Technical lead: directs investigation and mitigation work.
  • Communications lead: publishes internal, customer-facing, or stakeholder updates.
  • Scribe: records decisions, timestamps, commands, links, and owners.

One person may hold several roles during a small incident, but state that clearly. Escalate when impact crosses the documented severity threshold, when the responder lacks access or expertise, when mitigation has failed, or when a security, privacy, or data-integrity concern appears.

Use a predictable update cadence. For example, publish an initial note after impact is confirmed, then provide updates at an agreed interval or whenever severity, scope, mitigation status, or expected recovery changes. Each update should include current impact, actions underway, known uncertainty, and the time of the next update. Do not promise a recovery time that has not been established.

4. Mitigation and recovery

Choose the least risky action that can reduce user impact. Depending on the system, this may include rolling back a recent release, disabling a feature flag, shifting traffic, reducing load, restoring a dependency, or temporarily limiting an expensive operation.

  1. State the proposed action and expected effect.
  2. Identify the person responsible and the approval or change path required.
  3. Record the starting metrics and relevant configuration.
  4. Execute the change using the documented mechanism.
  5. Watch the success indicators and failure indicators for a defined observation period.
  6. Stop, revert, or escalate if the result is not as expected.

Do not treat a green deployment status as proof that the incident is resolved. Confirm recovery using user-facing indicators such as successful requests, acceptable latency, queue depth, background-job completion, and error-budget signals where applicable. The SRE error budget and SLO guide provides useful context for connecting incident decisions to reliability targets.

5. Handoff and closure

A handoff should let a new responder understand the incident without reconstructing the entire conversation. Include:

  • Current severity and user impact
  • Start time and important timeline events
  • What has been ruled out
  • Actions taken and their observed results
  • Open hypotheses and next checks
  • Active risks, temporary workarounds, and rollback status
  • Named owners for follow-up actions

Close the operational response only after impact has stopped, monitoring is stable, stakeholders have been updated, and follow-up work has an owner. If the cause remains unknown, close the incident as an active outage only when the service is stable; create a separate investigation task with a clear deadline.

What to double-check

Review these details before an incident occurs and during scheduled runbook maintenance:

  • Access: Can on-call responders reach dashboards, logs, deployment systems, cloud consoles, repositories, and incident channels without waiting for a manual permission grant?
  • Links: Do service ownership, dependency, dashboard, alert, rollback, and escalation links still resolve?
  • Commands: Are examples safe, scoped, and labeled as examples? Remove destructive commands that lack warnings and recovery instructions.
  • Alert meaning: Does each alert explain what it measures, likely user impact, and the first useful investigation step?
  • Time handling: Are timestamps recorded with a consistent timezone and format?
  • Secrets: Does the runbook avoid tokens, credentials, private keys, and sensitive customer data? Review the secrets rotation checklist when response procedures involve credentials or certificates.
  • Change history: Can responders identify recent releases, infrastructure changes, feature-flag updates, and dependency failures?
  • Recovery criteria: Is “resolved” defined with measurable signals rather than a belief that the system looks quieter?

Run a tabletop exercise using a realistic alert. Ask a responder who did not write the runbook to follow it, then record every missing permission, ambiguous term, stale link, and unexplained decision.

Common mistakes

  • Starting with root-cause analysis: A detailed theory is less valuable than confirming scope, protecting users, and reducing impact.
  • Leaving coordination implicit: Without an incident commander and scribe, several people may repeat the same work while critical decisions remain undocumented.
  • Overloading the runbook: A single document that contains every possible failure mode becomes difficult to scan. Keep the main sequence short and link to focused service procedures.
  • Using vague escalation rules: “Escalate if needed” does not tell responders when to act. Tie escalation to impact, duration, uncertainty, access, and security or data-integrity risk.
  • Communicating only at the end: Stakeholders need a reliable view of impact and next steps even when the cause is unknown.
  • Making unverified changes: Emergency does not mean undocumented. Record what changed, why, who approved it, and how it will be reversed.
  • Writing a blame-focused review: A blameless postmortem examines system conditions, controls, decisions, and incentives. It should produce practical improvements rather than a search for an individual at fault.

For the post-incident review, document the customer impact, detection quality, response timeline, contributing conditions, successful actions, failed assumptions, and a small set of owned follow-up items. Each action should have a measurable outcome and a review date.

When to revisit

Review the on-call runbook monthly or after every significant incident. Update it immediately when service ownership, escalation roles, communication channels, alert definitions, deployment workflows, access requirements, or rollback procedures change. Reassess severity criteria at least quarterly and before seasonal planning cycles, traffic changes, major launches, or other periods when failure impact may differ.

Make the review practical: test links, verify access, replay one alert, confirm that contact information is current, and remove procedures that no longer match the system. If your platform or deployment workflow changes, update the linked operational guidance as well; the platform onboarding playbook can help service teams keep ownership and infrastructure details visible.

Next action: copy this structure into your team’s runbook repository, replace each placeholder with a real service or link, assign an owner, and schedule a short tabletop exercise. After the exercise, publish the revised version as the operational source of truth and record the next review date.

Related Topics

#sre#incident-response#on-call#observability#reliability
C

Control Center Editorial Team

DevOps Guides Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.