The bug that returns success
Everyone assumes
Failures announce themselves. Something throws, something 500s, something pages you.
What we found
The expensive failures don't throw. They return a well-shaped empty object, die quietly between heartbeats, or drift one URL at a time — and every status check stays green while they do it.
The evidence
From our own incident ledger across several projects: a scheduled job that had been dead since a specific pull request, discovered only much later; API responses that were richly structured and completely empty; a front-end and back-end drifting apart until users were served 404s; and — the one that stings for a cost advisor — a 5× cost increase from a model change nobody noticed, because nothing was watching for it.
Why it happened
Success is the default return path. An empty result, a skipped step, and a correct-but-vacuous answer all look identical to a check that only asks “did this return without error?” Visibility of failure has to be designed in; it is not a property you get by adding logging.
What it costs you
Silent cost drift is the worst kind, because the meter keeps running while every indicator says fine. A 5× increase discovered at the end of the month is a 5× increase you already paid.
Check your own setup
Take one scheduled job you rely on. Ask how you would find out if it stopped running tomorrow. If the answer is “someone would eventually notice the output was stale”, it is already possible that it stopped.
Distilled from our own cross-project incident ledger.