Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
Disaster recovery plan: setting RPO and RTO, and what to test How to write a disaster recovery plan a small team can actually follow: setting RPO and RTO per system, writing the runbook, and the tests that prove the…
The database server will not boot. The provider says the disk has failed. Someone asks how long until the system is back, and how much data is gone. If the honest answer is "we will find out", you do not have a disaster recovery plan yet. You have backups, which is a good start, but not the same thing.
A disaster recovery plan is a short document that states, for each system, how much data you can afford to lose (RPO), how long it may be down (RTO), and the exact steps to bring it back within those limits. It names who does what and how the plan is tested.
RPO and RTO in plain terms Recovery Point Objective (RPO) is the maximum age of the data you restore. If your RPO is 24 hours, losing everything written since last night's backup is acceptable. If it is 5 minutes, a nightly dump cannot meet it, however reliable it is.
Recovery Time Objective (RTO) is the maximum time from the start of the outage until users can work again. It includes noticing the problem, deciding to recover, provisioning a machine, restoring data, repointing DNS and checking the result. Teams often quote only the restore step and then miss their RTO by hours.
Both are business decisions, not technical ones. Engineers can say what each target costs; the owner of the application decides what is acceptable. A fee-collection system during admissions and an internal wiki should not share the same numbers.
Setting targets per system Start with an inventory. List every system that matters, what it depends on (database, object storage, DNS, email provider, identity, secrets) and who owns it. Then sort them into a few tiers. Three is usually enough:
Tier 1: stops the organisation working or loses money or records when down. Typical targets are an RPO of minutes and an RTO of a few hours. Tier 2: painful but people can work around it for a day. An RPO of a few hours and an RTO of one business day is common. Tier 3: internal tools and reporting. A nightly backup and a restore within a few days is usually fine. Then match each target to a mechanism, because the target decides the technique. A 24-hour RPO is met with nightly logical dumps. An RPO of minutes needs continuous archiving: for PostgreSQL that means base backups plus WAL archiving with a tool such as pgBackRest or WAL-G, which also gives you point-in-time recovery. For MongoDB, it means a replica set with members on separate machines plus oplog-aware backups; replication alone is not a backup, because a bad delete replicates too. An RTO of under an hour usually means a warm standby you can promote, not a restore from scratch, because restoring a large database alone can take longer than that.
Be honest about what the cost buys. Each step down in RPO or RTO adds infrastructure and moving parts that also need monitoring and testing. If the owner asks for near-zero numbers, show them the standby, the replication and the extra test effort, and let them choose.
What goes in the plan Keep it short enough that someone tired and stressed will actually read it. We write one page per system, kept in version control next to the infrastructure code, with a printed or offline copy somewhere that does not depend on the systems it describes. A template we use, filled in with example values:
# illustrative entry; the values are examples
system: admissions-portal
owner: Registrar's office (decides on failover)
tier: 1
rpo: 15 minutes
rto: 4 hours
depends_on:
- postgresql 16 (primary on app-db-01)
- object storage bucket: admissions-uploads
- DNS zone and TLS certificates
- SMTP provider for notifications
backups:
database: pgBackRest, full weekly, diff daily, WAL archived continuously
files: nightly sync to off-site bucket, versioning on
config: git repository (compose files, NGINX, env templates)
secrets: password manager vault "DR", two named people with access
recovery_steps: runbooks/admissions-portal-restore.md
last_test: 2026-09-12, full restore, 2h 50m, RPO met (last WAL 4 min old)
next_test: 2026-12The runbook it points to is a numbered list of commands and checks, not prose. Each step says what to run, what output to expect and what to do if it differs. Include the boring details that stop recoveries in their tracks: where the encryption key is, which DNS records to change and their TTLs, how to issue a new certificate, and who can approve spending on a new server.
Add a contact and decision section. Who declares a disaster? Who tells users, and through which channel if email is down too? When do you stop trying to repair the old server and start recovering on a new one? Writing that cut-off down in advance (for example, "if the primary is not back within 45 minutes, begin restore") saves the hour that is otherwise lost to hoping.
Measure your real RPO, not the intended one Your actual RPO is the age of the newest backup that would restore right now. Monitor that directly instead of trusting the schedule. For PostgreSQL with WAL archiving, the statistics view shows when the last segment was archived and whether any failed:
SELECT last_archived_wal,
last_archived_time,
now() - last_archived_time AS archive_age,
failed_count,
last_failed_time
FROM pg_stat_archiver;Alert when archive_age exceeds your RPO, and when failed_count rises. Note that on a quiet database WAL segments fill slowly, so set archive_timeout (for example to 60 seconds) if you need a short RPO during low traffic. A fully idle database still archives nothing, so expect the age to climb overnight and either tolerate that window or check that writes are happening before you page anyone. For dump-based backups, alert on the age of the newest file in off-site storage, not on the cron job's exit code. A job that silently stopped running reports no errors at all.
What to test, and how often Testing is where most plans fall apart, so we use four levels, each catching different failures:
Automated restore check (weekly): fetch the latest backup from off-site storage, restore it into a disposable container and run data checks. This proves the backup is usable. Point-in-time recovery drill (quarterly for tier 1): restore to a specific timestamp, such as just before a simulated bad deployment. This proves the WAL chain is complete. Tabletop exercise (twice a year): walk through a scenario on paper with the people named in the plan. "The cloud account is locked. What now?" This finds missing credentials, unclear ownership and steps nobody knows. Full recovery test (at least yearly per tier 1 system): rebuild the whole service on fresh infrastructure from the runbook alone, ideally by someone who did not write it, and time it end to end. For a pgBackRest point-in-time drill on a test server, the restore itself is one command; the work is in checking the result:
sudo systemctl stop postgresql
sudo -u postgres pgbackrest --stanza=main --delta \
--type=time "--target=2026-10-01 14:30:00+05:30" \
--target-action=promote restore
sudo systemctl start postgresql
# Confirm the newest row is from just before the target time
sudo -u postgres psql -d admissions -At \
-c "SELECT max(created_at) FROM applications;"During a full test, record timestamps at each stage: outage declared, server ready, data restored, application up, DNS switched, users confirmed working. The total is your real RTO. Compare it with the target and write down what slowed you down. Common findings are database restores taking longer than expected as data grows, DNS TTLs set to a day, missing environment variables, and a firewall rule that only existed on the old server.
Test the scenarios that are easy to forget, too. Can you recover if the whole provider account is unavailable? Can you recover if backups were deleted by a leaked credential, meaning you rely on immutable or write-only copies? Can someone other than the usual engineer do it?
Keeping the plan alive Plans go stale quietly. A new database is added and never backed up, a password is rotated and the backup job stops, the person with vault access leaves. Tie plan updates to changes: any new service, database or provider gets a plan entry before it goes live, and every test ends with the plan being corrected. Review the targets with owners once a year, because what was tier 3 last year may now run payroll.
A practical checklist Inventory every system, its dependencies and its owner. Agree an RPO and RTO with each owner, grouped into tiers. Choose a backup or replication method that can actually meet each target. Write a one-page plan and a step-by-step runbook per system, with an offline copy. Store backups off-site with credentials that cannot delete them, and keep keys reachable by more than one person. Alert on backup age against the RPO, not only on job failures. Run weekly automated restores, quarterly point-in-time drills, tabletops and a yearly full recovery. Record the real RPO and RTO from each test and fix the gaps before the next one. If your last full recovery test was "never", or the plan lives in one engineer's head, that is the place to start. Pick your most important system, set its two numbers with the owner and run one timed restore. The result will tell you more than any amount of planning.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789