It is almost always a disk. Not a dramatic failure, not a hack, just a disk that quietly filled up, on a Tuesday night, while everyone was asleep.
This one was a distributor. Their database had been growing for years, as databases do, and the log file had a habit of ballooning during the nightly batch. For years there had been just enough room. Then one night there was not. The disk hit zero, the database stopped accepting writes, and the overnight run that loaded the next day’s orders failed silently. Nobody knew until the warehouse turned up at six and had nothing to pick.
The fix took twenty minutes once someone with the right access was awake and looking. The cost was a morning of a business that could not dispatch, a scramble of phone calls, and a very awkward explanation to some large customers about where their freight was.
What gets me, still, is how loudly the disk had been warning them. The free space had been trending towards zero for weeks. The graph, if anyone had been watching the graph, was a straight line heading for a cliff. It was all there, in plain sight, telling anyone who looked exactly what was going to happen and roughly when. Nobody looked, because the disk had always coped before.
The absence of a problem is not the same as the absence of a risk. It is just a risk that has not collected yet.
We go and read the graphs nobody is watching, for free, read-only, and tell you which cliffs you are heading for before you arrive. Most of the time, the server told you. It just told you quietly, weeks ago, in a place nobody was looking.
Tell us what is going on. A senior DBA reads every message and points you at the right next step, even when that is not us. A free read-only check will tell you whether this one is sitting in your estate.
Get pointed in the right direction