Aug 2026 · Bake-off №1
The half-cent answer.
One genuinely hard question. Four models, two vendors, prices from half a cent to $1.70. A $0.005 model matched a $0.36 model — and the $0.08 one was the dangerous answer in the room.
Why we run these
Nobody publishes a quality index you can trust for your work, so Scruuge won’t pretend to have one. A bake-off is the honest alternative: run the real task, publish every answer, show what each cost. The prompt is built so weak answers fail visibly — a question that can’t expose a bad model isn’t a test, it’s a demo.
The ladder
Ranked by answer quality across four traps. Prices are per answer. GLM’s is measured; the Claude figures are list-rate approximations.
1st
Opus
≈ $1.70
Deepest everywhere, and the only one with genuinely novel load-bearing ideas — a multiple-comparisons ledger, and naming no calibrated abstention as the real cost driver. Broke the word cap.
2nd
Sonnet
≈ $0.36
Clean, disciplined, zero factual slips. Sharpest single caveat: on pure ratio the friend has already been beaten.
2nd =
GLM-5.3-flash
$0.0051 measured
Genuine Sonnet-class density, with a credit-assignment argument nobody else made. One garbled fact. ~70× cheaper than Sonnet, ~330× cheaper than Opus.
4th
Haiku
≈ $0.08
Confident structure — and invented “real incidents,” got compression facts backwards, and produced a fake-precise verdict. 16× the price of GLM, and far worse.
Scruuge’s TLDR
Cheap didn’t mean bad and pricier didn’t mean safe. The cheapest model was excellent; the second-cheapest was the one that would have hurt you. Fit for the task is the axis — not price, in either direction.
Scorecard — the four traps
The traps were fixed before any model ran. Each demands committed architecture with failure modes, mechanical rather than hand-wavy reasoning, and the honesty to concede a point.
| Trap | GLM-5.3-flash | Haiku | Sonnet | Opus |
|---|---|---|---|---|
| Architecture commitmentreal contracts and failure modes, not a survey of options | STRONG"Proposal distribution / acceptance function" frame | MIXEDConcrete contracts — but cited invented specifics | STRONGTest-oracle independence rule; vision-to-number ban | STRONGESTSpan-offsets + content hashes; multiple-comparisons ledger |
| Mechanical depthwhy "usually right" ≠ "reliably right" | STRONGCompounding math, correlated errors, tail miscalibration | ADEQUATELong-tail story holds but shallower | STRONGErrors correlated with mechanism, not sampling seed | STRONGESTNo calibrated abstention as the biggest cost driver |
| Compression honestywilling to say "he's mostly right"? | STRONGCredit-assignment: perfect verification, zero gradient | WEAKFacts backwards — claims learned models still trail LZMA | STRONGOn pure ratio he's already beaten (cmix, nncp) | STRONGEST"Out-iterate, not out-think"; MAP-Elites archive |
| Disciplineword cap, marked uncertainty, factual care | MET~1,150 words; uncertainty marked. One garbled fact | MISSEDUnder cap, but padded with confabulated specifics | METUnder cap; flagged genuine uncertainty explicitly | OVER CAP~1,450 words — everything earned its place, but it broke the rule |
Where all four agreed
Four models, two vendors, zero shared context — converging independently. That’s the highest-confidence output of a bake-off, and it’s worth more than any single verdict.
- The model proposes, deterministic code disposes. A checker that never consults a model decides pass/fail. Self-verification is theater.
- Execution feedback beats everything. Run output against ground truth the model can’t edit, feed structured failures back. It turns generation into search.
- Majority voting is overrated. It cuts variance, not bias — and the errors that matter are systematic, so all N samples agree confidently on the same wrong answer.
- Never let the model free-write a number. Compute it in an engine, inject by reference. “Eyeball the chart” is a hallucination generator.
- The oracle must be independent. A model must not author or edit its own acceptance criteria — it will weaken them, not from deceit but because that’s the gradient.
Notable divergences
What only Opus produced. A multiple-comparisons counter, so the twentieth backtest hypothesis can’t report itself as the first. And the sharpest framing of all: the biggest cost driver is no calibrated abstention — the model can’t tell you which 20% is the bad 20%, so you can’t route only the risky cases to expensive checks.
What only the half-cent model produced. Compression has perfect verification but zero gradient — the benchmark hands back one scalar after an hour, so the propose-verify loop that works elsewhere has nothing to climb. Unique across all four, from the cheapest contestant.
Haiku’s failure is the instructive one. It didn’t fail by being vague — it failed by confabulating specifics: production incidents narrated as firsthand experience, compression facts inverted, and precision-cosplay (“beats him by ~2%”). This is the fluent-confident-wrong failure mode in the wild, and it is exactly what makes cheap the wrong thing to optimise for on its own.
Full answers — judge for yourself
A scorecard you can’t check is an opinion with a table around it. All four answers are published verbatim, in ladder order, so you can disagree with our reading. That’s the point.
Read all four answers in full →Want a bake-off on your task?
We’re testing the appetite for this before building it out. If you’d find it useful to see your own real task run across the tiers — scored, costed, every answer published — leave your email and we’ll tell you when it’s available. No charge for saying you want it.
Get Scruuge's AI Cost Watch
Cautionary tales and cost wins — when there's something real to say. No spam.
No spam. Unsubscribe any time — or email us and we’ll remove you.
Format and analysis: Scruuge. Prices are per-answer at list rates except GLM-5.3-flash, which is measured. One task, one grader, one run — this ranks these answers, not these models in general. That distinction is the whole reason we publish the answers too.