system-one · working notes · ~/code/system-one · September 2026
Kahneman called System 1 "a machine for jumping to conclusions." Typesafe built one — Jev answers typed questions about text with calibrated probabilities instead of generated prose — and published no paper on how it works. system-one (the name is Kahneman's own term for the faculty it tries to be) reconstructs that behavior from public evidence on small open models, on a Mac, one measured experiment at a time: a machine for jumping to conclusions, carefully. The end state is a two-tier lineup — a 2.5B quality model quantized to 8-bit and a 0.6B volume model, both frozen, calibrated, and drop-in compatible with the official SDK. This page is the result, the mechanism, and the story — including what broke.
Why it matters
The name is the thesis. Jev is named for the Jevons paradox — the economist W. S. Jevons's 1865 observation that when steam engines got more efficient, Britain burned more coal, not less: make a resource cheap enough and you don't save it, you find a thousand new uses for it. Machine judgment is that resource now. Cut a decision's cost from an LLM call — seconds, cents, a prompt to parse — to one forward pass at ~10 ms, and you don't automate the same decisions cheaper; you discover the millions of micro-decisions that were never worth automating at all.
Because most of what automation actually does is decide: route this ticket, flag this transaction, escalate this case, block this content, approve this claim. The output that matters is a branch condition, not text for a human to read. Speed is the visible benefit of a System One model — but four other properties are what make it fit for decision-taking systems where an LLM is the wrong tool:
Where this shape of model earns its keep: ticket triage and intent routing, fraud and claims operations, content moderation queues, compliance screening, escalation gates, and — increasingly — as the fast decision layer inside agentic systems, where a large model plans and a System One model adjudicates every small step. Eight of these patterns ship as runnable examples in the repo — each with the real output of five systems, from the bare backbone to pinned Jev itself, including the honest misses.
correct answers per example against authored intended answers — 24 questions, an illustration, not a benchmark
| Example | 2B base (no FT) | 0.6B FT | 2B FT | 2B FT q8 (served) | Jev 1.13.0 |
|---|---|---|---|---|---|
| citation-check | 0/2 | 0/2 | 2/2 | 2/2 | 1/2 |
| claims-intake | 3/4 | 3/4 | 4/4 | 4/4 | 4/4 |
| lead-scoring | 3/3 | 1/3 | 3/3 | 3/3 | 3/3 |
| llm-guardrail | 2/3 | 2/3 | 2/3 | 2/3 | 3/3 |
| model-routing | 2/3 | 3/3 | 2/3 | 2/3 | 3/3 |
| moderation | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| rag-rerank | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 |
| support-triage | 3/4 | 3/4 | 4/4 | 4/4 | 4/4 |
| Total | 17/24 | 17/24 | 22/24 | 22/24 | 23/24 |
The ladder reads left to right: fine-tuning is worth +5 questions on the same 2B backbone, 8-bit quantization costs nothing, and pinned Jev leads by one. The single questions each side misses are telling: Jev is the only system that catches the prompt injection; the fine-tuned tiers are the only ones that catch a citation that overreaches its source — which Jev waves through. Raw outputs, requests, and the scoring script are all committed in the repo.
The result
Two tiers came out of this project, each judged on a reserved test it had never influenced. First, the volume tier: after freezing the recipe, the original reserved partition — 1,048 questions never touched by any training, tuning, or selection decision — was spent exactly once on the 0.6B model:
final test, temperature-scaled
lower is better · scale 0–0.15
each point is a confidence threshold; hover for detail — the marked point is the natural operating range
Measured on the spent test's predictions. Thresholds are in-distribution numbers; they do not transfer to shifted workloads (a measured finding — see the ledger). Confidence is top-1 probability, not a probability of correctness.
Context: the development-set numbers are 82.0% / ECE 0.020, so the untouched test costs ~4 points and doubles the calibration error — a realistic in-family transfer gap. Earlier 3B experiments reached 91% argmax agreement with the real Jev on held-out sentiment, within 2–3 accuracy points of Jev itself. Served locally, a 12-question request answers in ~121 ms end-to-end.
MiniCPM5-2B-Base — development numbers, deliberately not comparable to the spent-test headline above
The one number that resisted every 0.6B intervention — score — moved decisively (MAE 0.564 → 0.435). Serving is measured too: a light 3-question request answers in 358 ms at BF16, 195 ms quantized. The dev win was later confirmed on a fresh reserved test the model had never influenced: +6.8 accuracy points over the 0.6B, CI [+4.5, +9.2] — see the timeline for the full story of that test, including the calibration criterion it narrowly missed.
the fused 2.5B winner converted with MLX — full dev battery, drift gates at three precisions, 16-case HTTP serving sweep
Dev accuracy 86.6% vs 86.8% BF16, confident-error rates unchanged. 4-bit was measured too: 1.3 GB, but −1.4 accuracy points and no speed advantage over 8-bit on this hardware — it earns no tier of its own. The adoption gate is now closed (decisions 27–28): temperatures were fitted fresh under the q8 weight identity (choice 0.96, noul 1.07, score 1.39 — the score correction was real), the calibrated battery held with confident errors improving to 1.2% at t≥0.95, and the official TypeScript SDK smoke test passed against the q8 server, which reports the temperature hash in its identity endpoint. q8 is the quality-tier serving configuration (decisions 28 and 30). SemIf independently flags the same class of issue on its quantized 27B bridge; our drift gate and identity-bound calibration measure it instead of footnoting it.
decision 30 — each tier judged once on a reserved test it never influenced
| Tier | Model | Weights | Unbiased result | Light request p50 |
|---|---|---|---|---|
| Quality | MiniCPM5-2B, fine-tuned, fused, 8-bit | 2.5 GB | +6.8 pts over volume tier on the fresh out-of-family test, CI [+4.5, +9.2] | 195 ms |
| Volume | Qwen3-0.6B + LoRA, frozen recipe | 1.4 GB + 33 MB | 77.6% / ECE 0.049 on the in-family reserved test | 77 ms |
Recorded honestly: the quality tier's confirmation gate asked for calibration error ≤ 0.08 on the fresh test and measured 0.0803 — a miss by 0.0003, inside the estimator's own noise, applied as written. The adoption therefore rests on an explicit judgment entry citing that near-miss, not on the pre-registered gate. The two tests use different data (in-family vs out-of-family), so the tiers' absolute numbers are not comparable to each other. Confidence thresholds for both tiers are valid in-family only until shift-robust calibration lands — the measured reason is in the ledger.
The key idea
An LLM and a System One model can be the same network — what changes is what you take from it. An LLM generates text token by token and you parse what comes out. Here, nothing is ever generated: one forward pass, then read the probability the model assigns to each allowed answer, directly from its output layer.
The frozen recipe
The story
Every step below is a frozen-protocol experiment with grouped-bootstrap confidence intervals; amber dots are the setbacks, kept on the record because they changed the plan.
POST /v1/calibrations with ~100 labeled decisions from a deployment's own traffic. The server scores them through the exact serving path, fits the temperatures, and hands back an identity-bound artifact — with a verdict per primitive that says whether a temperature is even the right tool, because the experiment showed that half the time it isn't. Unknown workloads fail closed; fits reload at startup under the same fail-closed provenance as everything else. The served fit matches the offline experiment pipeline to four decimals.The honesty ledger
The ecosystem
| Project | Backbone | Trained? | Calibration measured? | Distinguishing trait |
|---|---|---|---|---|
| jeff | 400M encoder | no (zero-shot) | global temperature | API server packaging |
| kev | 0.8B–9B family + pointer head | yes, incl. targeted second passes | yes, temp-scaled | 9B trails Jev by 3.5 pts on their suite |
| Nimble | 9B logit readout | yes, contrastive data | no | minimal-pair curation |
| Laya | 421M bidirectional encoder | yes, RLCD (published) | partially | 13ms latency; 512-token ceiling |
| openjev | DiffusionGemma 26B-A4B | no (strong base, zero-shot) | unknown | diffusion readout; free hosted tier; images |
| SemIf | Qwen3.5-4B frozen (also 0.6B, MiniCPM) | no (frozen checkpoints) | post-hoc, per workload | 4 runtimes incl. browser WebGPU; per-row provenance |
| system-one (this project) | 0.6B + 2.5B logit readout | yes, frozen protocol | primary metric, with CIs | teacher distillation + audited retractions |
Jev itself (~83% MMLU-Pro per black-box probes, 64k context) implies a far larger backbone — the competition here is on calibration, cost, and self-hosting, not frontier reasoning. SemIf is the closest cousin — same readout, same reproducibility instincts, even the same backbones — with one decisive difference: it never trains. Its frozen 4B needs temperature 2.5 to drag calibration error from 0.21 to 0.07 post-hoc; our fine-tuned models fit temperatures of 1.02–1.16 because calibration is trained in. That gap is this project's whole thesis.
Roadmap · as of September 2026
Snapshot at v0.1.0 — the living roadmap is docs/ROADMAP.md in the repo.
POST /v1/calibrations fits a workload's temperatures from ~100 labeled decisions through the exact serving path; requests opt in per call, unknown workloads fail closed, and the artifact is identity-bound and reloadable at startup like every calibration in this project. Each fit ships its own verdict — apply, neutral, structural warning, or degenerate — so the endpoint tells you when a temperature is the wrong tool. Verified end-to-end on the q8 tier: the served fit matches the experiment pipeline to four decimals.The measured 16–19% → 3–4% confident-error repair, one HTTP call away from any deployment.