A 149.6M-parameter decision model, trained in 8.8 hours on a $300 laptop GPU, that beats a 421M model on its own benchmark โ and ships a calibrated confidence channel nobody else has.
Benchmark: LocalLLaMA/typed-decisions, test split, N = 2,000 decisions. Read exactly once.
Against the model it replaces, on the same harness and the same split.
Accuracy went from 26.10% โ below uniform random โ to 77.10%, a 2.95× jump, clearing the TF-IDF floor of 66.10% by eleven points. Option-order flips run 36% below Kev's published rate.
| Model | Params | Accuracy | Brier | ECE (dist.) | ECE (conf.) | Flip rate |
|---|---|---|---|---|---|---|
| Verdict 1.0 baseline | 149.6M | 26.10% | 0.5851 | 0.4209 | โ | โ |
| TF-IDF + logistic regression | โ | 66.10% | 0.1520 | 0.0207 | โ | 0.00% |
| Jev 1.13.0 โ | ~150M | 72.70% | 0.1480 | 0.1440 | โ | โ |
| Laya (ModernBERT-large) | 421.3M | 76.60% | 0.0660 | 0.2140 | โ | โ |
| Kev-0.5B (Qwen2.5) โก | 500M | โ | โ | โ | โ | 7.41% |
| Verdict 2.0 Base | 149.6M | 77.10% | 0.0636 | 0.1513 | 0.0144 | 4.76% |
โ Vendor-published figure; not independently re-measured here. โก Kev's published flip rate on the identical option-order perturbation. Kev has never been evaluated on typed-decisions, so only its stability number is comparable.
Soft-labeled benchmarks hide a real mathematical conflict, and it explains the entire table above.
The gold labels are not 0/1. They are expert-panel distributions, and the panel's average top probability is 0.659 โ measured human doubt. Meanwhile the model is right 77.1% of the time. That gap is the trap.
Your top probability sits near 0.62 while you are right 77% of the time. Calibration error against correctness blows out. This is Laya: Brier 0.066, ECE 0.214.
You now track your hit rate, but you have stopped matching the panel and your Brier inflates. This is Jev: ECE 0.144, Brier 0.148.
One vector cannot do both jobs, because they are different questions. Verdict 2.0 ships two channels and refuses the trade.
| Channel 1 โ Distribution | Channel 2 โ Confidence | |
|---|---|---|
| Question | What does the expert panel believe? | Is my answer correct? |
| Mechanism | Marker-pointer logits + per-bucket temperature | CorrectnessHead MLP over distribution shape |
| Fit on | Calibration fold, 820 held-out decisions | Calibration fold, out-of-fold predictions |
| Scored by | Brier 0.0636 | ECE 0.0144 |
The confidence head never sees the gold label as an input โ only the shape of the prediction: top probability, margin between first and second, normalized entropy, option cardinality, question type. It is trained on decisions the encoder was never fit on.
And the distribution channel's calibration error turns out to be almost perfectly explained by the panel-doubt gap itself:
accuracy (0.7710) − mean distribution confidence (0.6197) = 0.1513 reported ece_distribution = 0.1513
Exact to four decimals. The distribution channel is not disordered โ it is correctly shaped and deliberately offset, because it reports the panel rather than its own hit rate. That is what Channel 2 is for.
A 2-layer MLP reads the bidirectional hidden state at each [MASK] marker and emits one logit per option โ simultaneously, inside a single attention pass. No generation loop, no decoding, no parse step.
Choice gets soft cross-entropy plus Brier. Score adds Ranked Probability Score, so predicting level 1 when the truth is level 5 costs more than predicting level 4. Result: MAE 0.2409, and 99.0% land within one level.
On 30% of steps, a twin batch with shuffled options is scored and the symmetric KL between the two predictions is added to the loss. Flip rate lands at 4.76% against Kev's published 7.41%, with p90 swing compressed from 0.2486 to 0.0915.
One temperature per (question type, option cardinality) bucket for the distribution channel; the correctness head on out-of-fold predictions. The test vault is touched by neither step.
At 149.6M parameters (<600 MB unquantized, ~150 MB INT4), openJev-verdict-2.0 runs locally in client browsers via WebGPU and ONNX Runtime Web. Zero network latency, zero server bills, and zero data exfiltration.
The confidence head achieves 0.7664 AUROC on 2,000 held-out test decisions. In automated pipelines, gating decisions at 80% coverage lifts accuracy to 83.44%; gating at 60% coverage delivers 89.00% accuracy, and 50% coverage delivers 91.40% for reliable human-in-the-loop escalation.
No cluster. No A100. No cloud bill. One consumer laptop GPU, overnight.
The result outperforms a 421M-parameter model on the benchmark that model was published against.
| Workflow | Decisions | Accuracy |
|---|---|---|
| Invoice processing | 500 | 81.2% |
| Customer service | 500 | 78.4% |
| Security incidents | 500 | 75.6% |
| Agent trace observability | 500 | 73.2% |
ModernBERT-large was the original plan. We stopped at Base, and the reason is on the record rather than after the fact.
The runbook pre-registered the gate for the shipping model before any training started: dev accuracy ≥ 0.760 and dev Brier ≤ 0.120. Base delivered 0.785 and 0.0639 โ it cleared the bar that had been set for the large model.
Base had also converged. Dev accuracy across the final four epochs: 0.7775 → 0.7850 → 0.7762 → 0.7825, oscillating inside half a point. Against that, Large needs 8-bit AdamW and gradient checkpointing to fit in 6 GB at all, and would run an estimated 14โ18 hours on this hardware.
The test split was treated as a vault.
Reference floors are published alongside the results, not omitted: uniform random 29.85%, majority label 48.35%, TF-IDF + logistic regression 66.10%, gold-distribution oracle 98.45%.
| File | What it holds |
|---|---|
| reports/verdict2_base_test.json | Official test-vault receipt, all metrics |
| artifacts/verdict2-base/dev_metrics.json | Out-of-fold dev metrics |
| reports/reference_floors.json | Baseline floors on the same split |
| verdict2/train.py | Training, temperature fitting, confidence head |
| verdict2/evaluate.py | The single test read, with the anti-leak gate |
| RUNBOOK.md | Exact commands to reproduce end to end |
Every number on this page traces to a file in the repository.