Where System One models win — and where they don’t.

What it is.

A System One model returns a typed judgment — a choice from a fixed set, a yes/no probability, or a position on a described scale — and never text. Ask it whether a ticket is urgent and it answers with a number, not a sentence about urgency.

It’s trained for calibration: “outcomes assigned a probability of 0.8 should occur about 80% of the time” (TypeSafe’s machine-learning primer). A confidence number that means what it says is the whole point — code can set a threshold on it and trust the threshold.

Try it. A System One model decides which class fits your task.

Describe an AI task below. TypeSafe’s Jev reads it and answers five questions in one call — what class of model fits, whether the output has to be prose, how often it runs, whether it has to be fast, and how bad a wrong answer would be. The card below is that answer, plus the receipt: how long it took and what it cost.

Where it wins.

Bounded decisions, at volume or at typing speed. Classify, route, rank, verify, pick one of a fixed set — anywhere the answer is a label or a probability, not a paragraph. The confidence number is what makes this useful in code: set the threshold once, and send only the low-confidence cases to a person or a bigger model.

One outside benchmark, on a computer-use decision loop, is the author’s own — not ours, and not independently verified — but it’s the clearest public number we found: 0.13–0.38 s and about $0.0002 per decision for a System One model, against about 5.2 s and about $0.032 for a frontier model doing the same reasoning unaided. That gap is the shape of the whole argument: most of it is speed and price, and it only holds for the bounded slice of the work.

What we measured ourselves.

We ran Jev against real work in two internal tools, not vendor demos. Both are dated, both include the parts that didn’t work.

On the 12 companies we analysed, Jev matched a five-judge LLM panel’s agreement with human-labelled verdicts, at a fraction of the cost and a fraction of the wait. The honest half of the same test: it did not replicate on four held-out companies — one gained 0.71 over the deterministic baseline, one lost 0.63, two tied. With about eight labels per company, a single company swings the whole result. The finding is real on what we measured and unproven beyond it.

One question we asked Jev — does this test prove the feature behaves, or just that it exists — came back “behavior” 95.1% of the time. A question that answers the same way 19 times out of 20 isn’t discriminating between cases. Our read: the question was badly shaped, not that nearly every feature is fine. We couldn’t settle it either way — when a person tried to label the same pairs from a test’s title alone, they answered “can’t tell” on 18 of 20, so this run stays marked unvalidated.

Where it loses.

• • It can’t write. No prose, no draft, no explanation — that’s a different class of model entirely.
• • It isn’t perfectly repeatable on borderline cases. TypeSafe’s own consistency cookbook found raw agreement of 90.8% across repeated runs of the same rubric — labels flip on close calls. A 0.60 confidence floor (answer “uncertain” below it) raised that to 99.2%, with automatic labels on 74.2% of answers. The floor is the fix, not the model getting more consistent on its own.
• • A well-formed answer can still be wrong. Calibration means the confidence number is honest, not that the model is right.
• • It’s text-only and English-first. Other languages are accepted at lower accuracy; images and audio aren’t.
• • A quality judge still needs human labels to prove itself. The matching result above means something because a person labelled 122 pairs first; the audit result never got that far, and so it proves nothing yet.

How to use it.

1. Decompose the task, then combine the answers in code — not in one giant prompt.
2. Keep the confidence threshold in code, not in the prompt, so you can tune it without a new model call.
3. Let code find the candidates; let the model pick among them.
4. Send the middle band — the answers near the threshold — to a person.
5. When confidence is low, fall back to a coarser answer rather than forcing a guess.