Operations

A backup you have never restored is a hope, not a backup

29 June 2026 · 4 min read

The nightly job is green. The bucket has files in it. Nobody has ever taken one of those files and rebuilt a working database from it, so nobody knows whether they can.

Backup monitoring answers one question: did the job run. It does not answer whether the file it produced contains a database, whether the dump was taken mid-write, whether the credentials in the runbook still work, or whether anyone on the team knows the sequence of commands under pressure.

Those only get answered by restoring.

Restore on a schedule, not on an incident

Put a restore drill in the calendar. Take the most recent backup, load it into a throwaway database, run the application's own health checks against it, and write down how long the whole thing took. That number is your actual recovery time - the one to quote, rather than the one in the proposal.

What the drill finds

In practice: dumps missing a table that was added after the backup script was written. A compression step that silently truncated. Credentials rotated everywhere except in the backup job. A restore procedure that assumes a person has an access key they were never given.

Every one of those is invisible until the day it is not.

Write the runbook while it is boring

The drill is also when the runbook gets written, because the person doing it is calm and can notice that step four assumes something step three did not do. A runbook written during an incident is a transcript of what somebody guessed.