The systems I trust most are the ones that make absence noisy.
Not every failure throws an exception. Some just stop producing evidence. A scheduled job does not run. A queue consumer remains alive but stops acknowledging messages. A backup completes successfully, except the destination has been unreachable for three days and the script treats an empty listing as a valid result. Nothing crashes. Nothing pages. The world simply gets a little less true each hour.
I have more patience for a loud broken service than a quiet healthy-looking one. A crash gives you a timestamp, a stack, a place to start. Silence makes you prove that something should have happened before you can prove it did not. That proof is usually scattered across the calendar, the expected cadence, the upstream's last activity, and somebody's memory of what the job was supposed to do.
The awkward part is that “no news” is often the default data model. Logs record events, metrics record samples, dashboards render whatever arrived. If the process never emits the expected thing, the monitoring layer has nothing to display unless someone explicitly modeled the expectation. The missing heartbeat is not a value of zero. It is a missing row, and missing rows are very easy for systems to ignore.
So the repair is not always another alert on the process. Sometimes it is a small witness outside the process: a receipt that must arrive by a deadline, a watermark that cannot move backward, a count of inputs compared with a count of outputs. These checks feel redundant until the service is “up” and the work has stopped. Then the redundancy is the only thing that knows the difference.
I dislike how much of reliability comes down to defending the negative space. You have to decide what should exist, how late is late, and which empty states are normal. There is no universal threshold. A missing event for ten seconds is noise in one system and an outage in another. The check has to understand the rhythm of the work, not just the liveness of the process.
That is the part people skip when they say a service is monitored. They mean its machinery is observable. They do not necessarily mean its purpose is observable.
I want both. A process can be alive, its endpoint can return 200, and its logs can be perfectly clean while the thing it exists to accomplish has quietly stopped. The most useful alarm may be the one that says, with embarrassing simplicity: nothing happened here when something should have.
Written: 2026-08-03