War story

The certificate nobody owned, and the Tuesday it expired

· Adrian Sullivan

A certificate expired on a Tuesday and an application stopped talking to its database. No deploy, no patch, no change request. The connection just died at a time the calendar had picked months earlier. This is a composite story. I have watched some version of it more times than I would like.

The shape is always the same. Someone stood the system up years ago and did the right thing at the time. They installed a certificate so the app and the database could trust each other and encrypt the traffic between them. Then that person moved teams, or moved on, and the certificate kept working. Working things do not ask for attention. The expiry date sat in the certificate the whole time, patient, waiting.

On the day, the symptoms looked like anything. The app threw connection errors. The first guess was the network. The second guess was the database being down. It was not down. It was up and fine and quietly refusing the handshake. People restarted services that did not need restarting. Two hours went into ruling out everything that was not the cause, because nobody thought to check a date that had been knowable since the day the thing was built.

Here is the part that stings. This was the most predictable outage you can have. A certificate tells you exactly when it will fail, down to the minute, the moment you install it. The information was never hidden. It was scattered across a dozen servers that nobody read together, in plain sight, a small disaster sitting on a fixed timer. The server had been telling you the date all along. There was just no calendar entry, and no name against it.

The lesson is dull and it is the whole point. The technical fix is trivial. You renew the certificate. The reason it became an outage is not technical. It is that nobody owned the list of what expires and when. No owner means no warning, and no warning means you find out from a failed login screen instead of a reminder. When I look across an estate, the expiry dates are usually there to be read. The watching is what is missing.

A free read-only health check finds those dates before they find you. Fifteen minutes, no changes to anything, and you get a graded plain-English report that lists what is expiring and roughly when. Better to read it now, on an ordinary day, than to learn the date the hard way on some random Tuesday.

Free health check

Want to know if this is sitting in your estate? We run a read-only check and hand you a graded report in plain English.

Get your free health check

← All posts