Certificates are the thing on your website that stops the browser showing a warning. They expire every ninety days now, so every organization renews them automatically and stops thinking about it, which is correct — until the automation quietly stops working and nobody finds out for two months.
We nearly shipped exactly that. The interesting part is that nothing was broken. Two decisions, each correct on its own, combined into a failure that would have surfaced sixty days later as a browser warning on a live site.
Two right answers that cancel out
There are two common ways to prove to a certificate authority that you control a domain. One has the renewal tool answer the challenge itself, which means it needs the web port to itself, which means the web server has to be stopped for a few seconds while it runs. The other serves the challenge through the running web server, which means the web server must stay up.
Every certificate on our estate used the first method. So a step had been added, sensibly, that stopped the web server before renewal ran. It was correct. It had worked for a year.
Then we added one certificate using the second method. Also correct, and better: no downtime. But the stop step was not aware of it, and it runs on every invocation of the renewal tool rather than only when the first method needs it. So the sequence became: shut down the web server, then ask a certificate authority on the other side of the world to fetch a file from that web server.
Neither decision was a mistake. The stop step was right for the certificates that existed when it was written. The new method was right for the certificate we added. The failure lives in the combination, which is where a lot of infrastructure failures live and why they survive review — each change is defensible on its own.
The error message was true, which is what made it hard
The certificate authority reported that the connection was refused. That was not a misleading error. It was precisely accurate: at the moment it tried, the port really was closed, because our own tooling had just closed it.
Everything we checked to explain that refusal was healthy. The firewall was configured to drop unwanted traffic rather than reject it, which produces timeouts rather than refusals, so it could not have been the cause. The intrusion-prevention system was blocking two brute-force attempts and nothing else. DNS resolved correctly and identically from every nameserver. File permissions were fine after one genuine fix.
Meanwhile every test we ran succeeded. We fetched the exact URL the certificate authority was fetching, from outside, and got the file back. The tests were passing because they ran in the gaps — the server was only down for the few seconds of each renewal attempt, and nothing we ran happened to land inside one.
The measurement that was wrong for a subtler reason
There was a second trap underneath. Every successful external test reached the server over IPv6, because that is what modern tools prefer when a domain has both kinds of address. We never actually proved the older IPv4 path worked, and the certificate authority was reporting its failure against the IPv4 address. We had spent an hour confirming something we had not tested.
That is worth generalizing. When a test passes, ask what path it took, not just what result it gave. A green check that exercised a different route than the failing one is not evidence, and it is the most convincing kind of non-evidence because it looks exactly like proof.
What finally settled it
We turned on request logging for the one address involved and ran the renewal again. Nothing arrived. Not a refused connection, not a failed request — no record of any attempt at all, because there was no server running to record it.
That single observation ended the investigation. Everything before it had been inference about what might be blocking a request; this was direct evidence that no request was being received. When something is behaving impossibly, the fastest route is usually to instrument the actual event rather than reason harder about it from the outside.
The fix, and the more important fix
The narrow fix took a few lines: the stop step now checks whether any certificate that actually needs the port is due for renewal, and leaves the server running otherwise. As a side effect, routine renewals no longer take the whole estate offline for a few seconds when nothing requires it.
The broader fix matters more. We added a check that looks at the outcome rather than the cause: does any certificate have fewer than twenty-one days left. Certificates last ninety days and renewal begins at thirty, so anything below that has already failed at least once, and there is still a fortnight to act. It does not care why. It catches this cause and the ones we have not thought of.
That is the transferable part. You cannot enumerate the ways an automated process might break, and this one was not on anybody's list. You can usually check whether it produced what it was supposed to. If your organization renews certificates automatically, the question worth asking is not whether the renewal job is configured correctly. It is whether anything, anywhere, would tell you if it stopped — before a client tells you first.
Working on something where this kind of thinking matters? Get in touch.