Reliability and Operations
Write incident runbooks that a teammate can follow
A useful incident runbook helps someone other than its author make safe decisions. Define entry conditions, observable results, escalation boundaries, and a rehearsal that exposes missing context before an actual disruption.
An incident runbook is a short operational guide for a recognizable failure. Its reader may have little context and limited time. The document should help that reader choose a safe next action, not recreate the author’s memory. Start with one recurring symptom and one supported service boundary rather than a large manual covering every possible outage.
Tell the reader when the guide applies
Open with user-visible symptoms, affected functions, and conditions that make this the wrong guide. Explain the access required without including credentials. Name the responsible role and the escalation route. A teammate should know whether to continue, request help, or stop before any change takes place.
Google’s incident management guidance recommends preparing documented procedures in consultation with incident participants.[3] Use that principle to include support staff and service owners in the review. Their questions often identify business context that a technical author leaves implicit.
Pair actions with observations
For each approved action, state its purpose, required authorization, expected result, and a stop condition. Keep diagnosis separate from changes. If a result differs from the expected observation, direct the reader to escalation rather than an improvised repair.
Describe how to preserve relevant evidence and record decisions. Link to controlled internal procedures in the actual operational document, but do not copy sensitive connection details into widely shared versions. The public outline of a runbook can explain responsibilities without exposing the system.
Hypothetical example: delayed warehouse labels
In this hypothetical example, warehouse staff report that label requests remain pending. The runbook first asks whether new requests are accepted and whether completed labels exist. It distinguishes a display problem from a processing backlog.
A teammate rehearses the guide and cannot identify who may pause new submissions. The team adds that authorization boundary and a customer update owner. The rehearsal reveals a documentation gap, not proof that a particular repair always works.
Review the handoff, not just the wording
Ask a colleague to work through a controlled simulation without verbal coaching. Note each place where the colleague has to guess. Use this checklist to decide whether the guide is ready.
Detailed instructions become expensive to review when the service changes. Excessively broad guides also hide the immediate decision. Keep background architecture in a separate maintained reference, and retain only what the responder needs in the runbook.
Assign an owner and review the guide after relevant changes or exercises. If a step becomes obsolete, withdraw or correct it rather than relying on an oral warning. The acceptance test is whether another authorized person can follow the document safely.
- Identify a clear entry symptom and an exclusion.
- Show the observation expected after each decision.
- Name who authorizes service changes.
- Provide a stop condition and escalation owner.
- Define who confirms customer workflow recovery.
