Loading data/benchmark.json …
Measurement notes: timings come from one Apple Silicon laptop under normal desktop load and vary between
runs — run minojev compare on your own hardware. Requests are scored one at a time; batching
and shared-prefix reuse favour the engine, while token generation pays a growing tail (p95) and needs a
parse step for anything richer than a single letter. Compact from-scratch models remain stronger
on the synthetic coordinate task (maze: 5/6 vs 2/6 solved against the frozen 1.7B head), which
is why the maze demo keeps the small model.
Per-source accuracy
| source | minojev | generative |
|---|
What “head training” means
1.7B frozen backbone 8000 requests cached features head only dev calibration
The backbone never updates. Every candidate path (state + question + option) is forwarded once and its final hidden state is cached; the decision head — a few million parameters — is then trained on those cached vectors with distribution losses, and temperatures are fitted on dev. In this run that was ~37 minutes and 4 GB of memory on a laptop, with no catastrophic-forgetting risk to the backbone. The result is a model that returns a typed, calibrated distribution with zero output tokens, instead of a sentence that has to be parsed.
minojev build-data --per-source 2500 --per-ood 800 minojev posttrain --backbone Qwen/Qwen3-1.7B --mode head \ --train data/general-train.jsonl --dev data/general-dev.jsonl \ --output-dir runs/general-head --steps 3000 --inference-dtype bfloat16 minojev compare --checkpoint runs/general-head/checkpoint \ --input data/general-test.jsonl --limit 120 --chat-template
Why it matters
Zero decoding
0 output tokens per decision. The generative baseline spends 2.5–3.5 tokens even when told to answer with a single letter.
No time to first token
There is no first token: the answer is a readout. The baseline waits ~100 ms before its first token and can stall for seconds at p95.
Calibrated confidence
Every answer carries a probability distribution, with dev-fitted temperatures and reported ECE — generation gives no probabilities at all.