Our mail backup had been running nightly for months and reporting success every time. Last night we checked whether anything could actually be restored from it. That had never been done.
It passed. The backup held 16,243 of the 16,329 messages on the live server, the 86 missing ones having arrived after the last run. We pulled one message out, and it parsed as a valid email with its headers and a 3,354-byte body intact. The whole exercise took about four minutes.
The point is not that it worked
The point is that until we looked, a working backup and a broken one produced exactly the same evidence. Both run on schedule. Both exit successfully. Both write a file that grows. Both send the same reassuring nothing. The difference between them only becomes visible on the day you need it, which is the worst possible day to discover it.
"The backup ran" is a statement about last night. "The backup restores" is a claim about the future, and running is not evidence for restoring. A job can complete successfully while writing a truncated archive, while skipping the one dataset that mattered, or while faithfully copying a database file that was mid-write and is therefore unopenable. The exit code is the same in every case.
What the test has to include
Opening the archive is not enough. Two things beyond the data itself decide whether a restore actually produces a working system, and both are easy to miss because they are not part of the file.
The first is ownership and permissions. Our backup files are owned by the same account as the live mail store. That sounds trivial and it is the thing most likely to go wrong: restore mail as the wrong user and every message is present, correct, and unreadable by the mail server. You get a directory full of intact data and a service that behaves as though the mailbox is empty. We checked this deliberately, because it is the failure that looks most like success.
The second is structure. A mail store is not just messages; it is messages plus the index files, the folder list and the delivery rules that make them navigable. Our backup carries those too. A restore that recovers the letters but not the filing cabinet leaves someone reconstructing years of folders by hand.
The generalization is that a backup is only as good as the smallest thing it omits. Whatever your system is, ask what has to be true besides the bytes for the restored copy to actually work, and check that specifically.
Know what a normal gap looks like
The 86 missing messages are worth dwelling on. That gap is correct: mail arrived after the backup ran. But if you do not know what a normal gap looks like for your own system, you cannot tell a healthy one from a broken one when you finally look. A backup missing 86 messages and a backup missing 8,600 both look like "some difference" if you have no baseline.
So write the number down. Ours is roughly a night's mail. Yours might be a day of transactions or an hour of files. The figure matters less than having one, because it converts a vague comparison into a check anyone can repeat.
The part we did not have
The data survived the test. The knowledge did not exist. There was no written restore procedure anywhere — no note saying which account owns the files, which service to stop first, what to copy where, or how to tell when it has worked.
That gap matters more than it sounds, because a restore is never performed on a calm afternoon by the person who set the system up. It is performed at speed, under pressure, quite possibly by someone who has never seen the inside of it, and often after the person who built it has left. A backup nobody knows how to restore is a backup that will be restored slowly and badly, if at all.
The procedure does not need to be elaborate. Three-quarters of a page covering which service to stop, what to copy, what owns it afterwards, and one sentence describing how you will know it worked. Write it during the test, while you are learning the answers anyway.
Do this one this week
Pick your most important backup. Restore one item from it — not the whole thing, one file, one mailbox, one record — into somewhere that is not production. Open it and confirm it is what it claims to be. Note who owns the restored copy and whether the service that needs it could actually read it. Write down what you did while you are doing it.
If it works, you have converted a belief into a fact and you now have a procedure. If it does not, you have found out on a Tuesday with everything still running, rather than during the incident that made you need it. Both outcomes are worth four minutes, and only one of them is available if you wait.
Working on something where this kind of thinking matters? Get in touch.