We audited our own monitoring recently and found a check that had failed every single day for four months. It watched the DKIM record on our mail domain — the cryptographic signature that tells other mail servers a message really came from us. If that record breaks, mail starts landing in spam and nobody tells you.
The check was red on all 78 runs we could still see. The DKIM record was fine the whole time. The check was asking for a selector that had never existed, from the day it was written.
The failure was the check, not the thing checked
Someone wrote the monitor, guessed at one value, and never confirmed the guess. The comment above it stated the selector as fact. It ran on a schedule, failed instantly, and mailed the result to an inbox where it looked like every other automated notice.
The interesting part is not the typo. It is what four months of red does to a reader. The first failure is information. The tenth is background. By the seventy-eighth, the check has trained everyone who sees it to look away, and it has done that specifically for the one topic it was built to watch. Had the DKIM record genuinely broken in that window, the alert would have arrived and been ignored, because it was indistinguishable from the noise the check had been producing since the day it shipped.
So the practical rule is narrower than "fix your alerts". A check that has never once passed is not monitoring anything. It is a permanently-lit warning light, and a warning light that is always on is furniture.
The same failure, inverted
The identical problem hides in silence. In the same audit we found a reporting job that ran on a schedule, exited successfully, and reported nothing to see. It was not reading anything: the log it summarized was readable only by the web server's own account, and the job ran as a different user. It could see the file existed and could not open it. An empty result and a healthy result looked exactly the same, so it would have reported quiet weeks forever, including after the traffic it was meant to summarize started arriving.
Another one: notifications that were being delivered correctly to a phone at a priority level the operating system does not display. The messages arrived. They were in the app. No notification was ever shown, so the channel looked dead while working perfectly.
In each case the system reported success. Nothing crashed, nothing paged anyone, and every dashboard was green. The failure was that green had stopped meaning anything.
A question worth asking about every check you own
Not "is it failing?" but "has it ever passed?" — and its mirror, "has it ever fired?" You do not need a monitoring platform to ask this. If a check has run for months and has never once produced the opposite result, one of two things is true: the thing it watches has been perfect for months, or the check cannot produce that result at all. From the output alone you cannot tell which, and the second is far more common than people expect.
For a small practice this is a short list and an afternoon's work. Your backup job that mails a report every Sunday: has it ever mailed a failure? Do you know what one looks like? Your antivirus console that has shown a green tick since installation: has it ever shown anything else, on any machine? The alerting address your firewall sends to: does anyone still read it, and did anyone leave the company since it was configured?
Test one of them on purpose. Rename a file the backup expects and see whether Sunday's report notices. Send yourself a message that should trip a rule. It takes ten minutes and it converts a belief into a fact. Almost every organization we look at has at least one check that has been reporting success into a void, and finding it costs nothing but the deciding to look.
What we changed
Two things, and both are cheap enough to copy. Every monitor we write now ships with a test that makes it fail — a deliberately broken input that must produce an alert. If we cannot make a check go red on demand, we have not demonstrated that it can go red at all, and we treat that as the check being unfinished rather than passing.
The second is that we now watch outcomes rather than causes. We had a certificate-renewal process that would have stopped working silently, for a reason nobody would have predicted: a cleanup step shut the web server down before the renewal could prove the domain through it. Rather than enumerate the ways renewal can break, we added one check that asks whether any certificate is running short. It does not care why. It catches the cause we found, and the ones we have not thought of yet.
None of this needs new software. It needs someone to ask, once, of each check they depend on: has this ever told me anything? The checks that cannot answer are the ones to fix first, because they are the ones you are already trusting.
Working on something where this kind of thinking matters? Get in touch.