It is always 2:39 in the morning. I do not know why. I have watched the clock on enough of these calls to believe it is a law of nature.
This one was a finance company, years ago. Month-end. The database that ran their billing had stopped, hard, and the on-call engineer had done the right thing: go to the backups, restore last night’s copy, be a hero before sunrise.
The restore would not complete. The backup file was there. It had been there every night for a year, green tick, job succeeded. It was also corrupt, and had been for months, and the job had never once tried to read it back. Nobody had asked it to.
We got them back. Not from that backup. From an older one, some log files, and a few hours nobody enjoyed. They kept trading. The lesson cost them a night, and it should have cost them nothing, because the warning had been sitting in plain sight the whole time.
A backup that has never been restored is not a backup. It is a folder you are emotionally attached to.
Here is the part that still gets me. They were not careless. They had backups, monitoring, a runbook, an on-call roster. They had done everything except the one thing that proves any of it works: restore the database, on a normal Tuesday, when nothing is on fire, and watch it come back.
We have all been there at 2:39am, coffee going cold, watching a progress bar that is lying to us. The whole point of what we do is so you do not have to. If you want to know whether your backups are real before the clock decides for you, we will check, for free, and tell you straight.
Want to know if this is sitting in your estate? We run a read-only check and hand you a graded report in plain English.
Get your free health check