The test suite said READY. Three of five recommendations didn't exist.
Everyone assumes
A green test suite means the product works. If something were badly wrong, something would be red.
What we found
A quality report graded our own calculator 92% — “core value delivery confirmed” — at a moment when three of its five model recommendations pointed at models that could not be bought, and four of fourteen prices were wrong, one of them by 4×.
The evidence
The report is dated and quotable: “Coverage: 92% · Status: READY”, blessing the calculator specifically as “two minutes, no account, real numbers.” A later check against the live model market found three recommended model IDs returned nothing at all. Both artefacts exist; neither needed hindsight to run.
Why it happened
Every test verified that a number was PRODUCED. None verified it was CORRECT. Reachability and correctness are different questions, and only one of them was being asked — so the suite could not fail on stale data, only on broken code.
What it costs you
A cost tool quoting a wrong price is worse than no cost tool: it is confidently wrong, and the person acting on it has no way to tell. Every downstream decision inherits the error with the confidence intact.
Check your own setup
Pick one number your dashboard shows a customer. Ask what would have to break for that number to go red. If the honest answer is “the code, but not the data” — you have a reachability test wearing a correctness badge.
Scruuge's own calculator, found and fixed August 2026.