Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
How to write incident runbooks your team will actually use A practical way to write incident runbooks: start from your alerts, use one fixed structure, make every command copy-pasteable, and test each page with…
The alert fires at 02:40: disk usage on the application server is above 90%. The person on call did not build this system. There is a wiki page called "Server notes" last edited two years ago, a Slack thread that mentions "the usual fix", and a colleague who is asleep. Twenty minutes go on working out where the logs are before anyone frees a single byte.
An incident runbook is a short page, linked from the alert itself, that tells whoever is on call how to confirm the problem, the safe first steps to reduce impact, the exact commands to diagnose and fix it, and when to escalate. The ones teams use cover one alert, scan quickly and have been tested by someone else.
Most runbooks fail for the opposite reasons. They describe a whole system instead of one failure, they are written as prose, and nobody has opened them since the day they were written. This post is how we write them for the services we monitor and operate.
Runbook, service page or postmortem? These three documents get mixed up, and the mix-up is why runbooks grow into essays nobody reads under pressure. Keep them separate.
Service page: what a service does, where it runs, who owns it, how to deploy and roll back. Read when you have time. Runbook: what to do when one specific thing goes wrong. Read during an incident, by a tired person, in a hurry. Postmortem: what happened in one incident and what will change. Read afterwards, and the main source of new and corrected runbooks. A runbook can link to the service page for background. It should never require reading it first.
Start from your alerts, not your architecture The common mistake is to sit down and write "the database runbook". That page tries to cover every way PostgreSQL can fail and ends up covering none of them well. Instead, list the alerts that can wake someone up and write one runbook per alert. If an alert has no runbook, either it is not worth paging for or you have found the next page to write.
For a small team, the first five are usually predictable: site down from the outside, disk filling up, certificate close to expiry, backup job missing or failed, and error rate or latency above normal after a deploy. Write those before anything else. Then look at your last few incidents and write the ones that actually happened.
Put the link in the alert, so nobody has to search. In Prometheus, the convention is a runbook_url annotation on the rule, which Alertmanager can include in the notification. Grafana alert rules have a runbook URL field that does the same job.
groups:
- name: host
rules:
- alert: HostDiskAlmostFull
expr: |
node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.10
for: 10m
labels:
severity: page
annotations:
summary: "Less than 10% disk free on {{ $labels.instance }} ({{ $labels.mountpoint }})"
runbook_url: "https://wiki.example.com/runbooks/host-disk-almost-full"Name the runbook after the alert. When the page title matches the alert name exactly, people trust that they are on the right page, and you can check for missing runbooks with a simple script that compares the two lists.
A structure that works at three in the morning Use the same headings on every runbook, in the same order. Familiar structure means the reader's eyes go straight to the section they need. This is the template we use.
# HostDiskAlmostFull
Last tested: 2026-09-12 by <name> Owner: <team>
## What this means
One or two sentences. What users notice, how urgent it is.
## Check it is real
One command or dashboard link that confirms the alert.
## Stop the bleeding
Safe first actions that reduce impact. No data loss possible.
## Diagnose
Commands in order, with what normal output looks like.
## Fix
One subsection per known cause. Mark destructive steps clearly.
## Confirm it is fixed
How to tell the alert will clear and users are fine.
## Escalate
Who to call, and when: e.g. not resolved in 30 minutes, or data at risk.
## Links
Dashboard, logs, service page, past incidents.Two parts of this template do most of the work. "Stop the bleeding" separates reducing harm from finding the cause, so the reader does not spend half an hour on root cause while users are still affected. And "Escalate" gives explicit permission to ask for help, with a time limit, which removes the hesitation that stretches many incidents.
Example: the disk is almost full Here are the diagnose and fix sections of a typical runbook for a Linux host running Docker. Every line can be pasted as it stands, and the comments say what to look for.
# 1. Which filesystem, how full?
df -h
# 2. Largest directories on that filesystem (-x stays on one disk;
# replace / with the mount point df reported)
sudo du -xh --max-depth=2 / 2>/dev/null | sort -h | tail -20
# 3. Usual suspects
journalctl --disk-usage
docker system df
sudo ls -lhS /var/log | head
# 4. Space held by deleted files still open by a process
sudo lsof +L1 | sort -k7 -n | tail
# Fix A: journal too large (safe)
sudo journalctl --vacuum-time=7d
# Fix B: unused Docker images and build cache (running containers are
# not affected, but older images you might roll back to are removed)
docker image prune -a -f
docker builder prune -f
# Fix C: deleted log still held open -> restart that service, not the server
sudo systemctl restart <service>Note what is not in it. There is no rm -rf, and no docker system prune --volumes. Volume pruning deletes every volume that no container references, so a database whose container was removed mid-redeploy loses its data. Before Docker Engine 23.0 that included named volumes, not only anonymous ones. If a cause needs a destructive step, the runbook says so in plain words, says what will be lost, and points to the escalation section first. The longer-term fix, such as log rotation limits or a bigger disk, goes in the postmortem, not the runbook.
Write for the person who did not build it The reader is competent but does not know your system and is under stress. Write for that person, not for yourself.
Commands, not descriptions. "Check the logs" is useless. "docker logs --since 15m api" is a step. Show what normal looks like. A command without expected output leaves the reader unable to tell good from bad. Use real names: hostnames, container names, database names, dashboard links. Placeholders only where the value genuinely changes. One action per numbered step. Decisions as plain if/then lines: if the queue is above 10,000, go to Fix B. Label destructive steps in capitals and put a check before them, such as confirming the latest backup exists. No secrets on the page. Say where the credential lives in the password manager, and who can grant access. Assume access may be the problem. Include the break-glass route if SSO or the VPN is down. Keep each runbook short enough to read on a phone. If it scrolls for pages, it probably covers more than one alert and should be split.
Keep them true: testing and upkeep A runbook that is wrong is worse than none, because people follow it. Three habits keep them accurate.
First, test each new runbook with someone who did not write it. Give them the alert text and a staging environment, and watch silently. Every place they hesitate, ask a question or open a different tool is an edit. It usually takes under an hour per runbook and finds more gaps than any review by the author.
Second, update during or straight after every incident. The person who used the runbook fixes what was wrong while the details are fresh. A postmortem action that says "update runbook" without a link to the change is not done.
Third, put a last-tested date at the top and review anything older than six months. Infrastructure changes quietly: a container is renamed, logs move to a different path, a dashboard is rebuilt. A small scheduled game day, where you trigger a known failure on staging and work through the runbook, catches this drift before an incident does.
Over time, look for steps that are always the same. A diagnosis block that every reader runs in full is a good candidate for a script, and a fix that is always safe may be worth automating so the alert never needs a human. Keep the runbook anyway, describing what the automation does and what to do when it fails.
Runbook checklist Every paging alert links to a runbook with the same name. Each runbook covers one failure and follows the same headings. There is a safe first step that reduces impact before diagnosis. Commands are copy-pasteable, with expected output shown. Destructive steps are labelled and preceded by a check. Escalation names a person or rota and a time limit. No secrets on the page, and a route in if normal access fails. Tested by someone who did not write it, with a last-tested date. Updated after every incident that used it. If you have alerts but no runbooks, start with the alert that fired most in the last three months and write that page this week. If you have neither, the alerts come first: a runbook only helps when something tells a person to open it.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789