The small model didn't fabricate. It stopped citing.

Can a 2.6B model hold down a real agent job? Not a demo, not a chat window: the web-research role in my personal assistant stack, where it runs an agent loop with search and page-extract tools and has to come back with a grounded, cited report. I benchmarked it against the 26B-class incumbent under a pre-registered decision rule. It was disqualified. The reason it was disqualified is the interesting part, and it was not hallucination.

The setup

The stack runs on the Strix Halo box: everything local via llama.cpp behind an OpenAI-compatible gateway with llama-swap routing models per role. The assistant itself is a Telegram bot, and its privacy design is the part worth describing. The assistant container lives on an internal-only network with no default route; its only egress is a proxy that allowlists the Telegram API and the local LLM gateway. It cannot browse the web at all. All web work is delegated to a research sidecar: a separate container with real egress, running its own LLM agent loop with web-search and page-extract tools against the same local gateway. Outbound queries pass a PII-rewrite gate before they leave the assistant's zone. The assistant never touches the internet; the sidecar never sees the conversation, only the rewritten research query.

The benched role is the model behind that sidecar's agent loop. The incumbent is gemma-4-26b-a4b, the assistant's main chat model: Google's April Gemma 4 MoE, 25.2B total parameters with 3.8B active, Apache 2.0, and currently ranked first on the FACTS Grounding leaderboard, ahead of several frontier models. The challenger is Liquid AI's LFM2.5-2.6B at Q8_0, all of 3 GB. It was eight days old on bench day, and it is marketed for exactly this job: the release is titled "Deploy Agents Everywhere," the vendor claims it leads instruction-following and tool-use benchmarks against models up to 4x its size, and their launch numbers cite 113 tok/s decode on the Ryzen AI MAX+ 395, the precise chip in this box. If any small model should hold this role, it is this one. (Liquid's own model card does add one honest caveat: not recommended for knowledge-heavy tasks.)

Why audition a 2.6B at all

Not memory. When the sidecar uses the same model as the chat assistant, the weights are already resident and the marginal RAM cost is roughly zero. The real motivation is residency-independence: this box gets switched between large models, and when the chat model is swapped out, research built on it stops working. A pinned 2.6B stays resident through every switch. Research that always works, regardless of what the box is currently serving, is worth something.

And the challenger had a record. Two days after its release it was already running the research role in production, and it ran it for about 2.4 days: 1.46M prompt tokens, 178K generated, zero tool-call parse failures, zero malformed JSON, zero reasoning leaks (llama.cpp's dedicated LFM2 tool-call parser earning its keep), ~71 tok/s decode at Q8_0. Mechanically flawless. I reverted it anyway, because its output in a different role (scheduled monitoring jobs) showed hallucinated facts and protocol violations. That revert was a judgment call, made on vibes, without a controlled measurement.

The DeepSeek post established that fitting on the box is not the bar. This bench exists because running cleanly is not the bar either. The bar is: does the work hold up, and you cannot answer that from an impression.

The bench

The design borrows from the eval literature (RAGAS-style reference-free faithfulness, ALCE-style citation support, FActScore's atomic-fact precision, MT-Bench pairwise judging with position swaps) and rejects the off-the-shelf harnesses, because the "model under test" here is a live production container whose arm-switching is a config mutation with quiescence requirements. Both arms would have become custom script providers anyway; the harness would have added nothing. The result is ~1,100 lines of Python, TDD on the pure logic, and an adversarial critique pass on the methodology draft before any data was collected.

The shape:

• 29 queries mined from real usage: current events, local venues, weather, technical research, comparisons, niche site-specific lookups. Fourteen are time-sensitive. Three are fabrication traps: topics with near-zero web coverage where an honest model reports scarcity and a fabricator invents entities (a term that exists only in a private spec, a plausible-sounding but unpublished benchmark score, a hyper-local news fact). I am keeping the actual trap topics out of this post so they stay reusable.
• 116 live research runs: 29 queries × 2 arms × 2 repeats, arms run back-to-back minutes apart against the same live web, arm order randomized per trial by seeded RNG. This ran against the production stack, with the production cron paused and restored in a finally block, quiescence enforced between trials, and every phase resumable.
• Tier 1, deterministic: extract every cited URL, then resolve it from inside the sidecar's own container, using the same egress and DNS the agent used. NXDOMAIN counts as fabricated unconditionally; a non-existent domain cannot have been read. Template artifacts (example.com, placeholder braces) count as fabricated. A deep-link floor prevents a cite-only-safe-homepages strategy from passing. Protocol compliance per report: non-empty, English, at least one citation, no refusal boilerplate.
• Tier 2, judged: pairwise preference per (query, repeat) by gpt-oss, a model family disjoint from both contestants, since letting the incumbent judge itself invites self-preference bias. The judge sees both reports plus the Tier-1 URL verification results injected as annotations; without that, an LLM judge happily rewards confident citations it cannot check. Every pair is judged twice with positions swapped, and the verdicts must agree or the pair scores as a tie. Separately, an ALCE-style citation-support check: fetch up to three cited pages per report and ask the judge for strict sentence-level entailment, with dead and fabricated citations counted as unsupported so a model cannot improve its support rate by having its bad links excluded.
• A measured noise floor: 15 control pairs, the same model against itself, judged identically. The cross-arm signal has to beat the judge's own same-model noise or the verdict is inconclusive.
• A pre-registered decision rule, fixed before data collection: the challenger is promoted only if it passes a fabrication gate (zero invented domains, dead links ≤5%, plus a manual read of every trap report), a protocol gate (≥95% report compliance), a quality gate (≥45% of decisive judge outcomes, explicitly signed in advance: speed does not buy a clearly worse answer), and a conclusiveness check (noise floor cleared). Statistics at query level: sign test plus Wilson 95% CI on the win share.

The results

Verdict first: LFM2.5-2.6B disqualified on the protocol gate. The quality gate also failed. Incumbent retained.

At the judge: 57 cross-arm pairs, position-consistency 79% against a 60% floor. At query level, 12 decisive wins for the incumbent, 5 for the challenger, 12 undecided. Challenger's decisive win share: 29%, Wilson 95% CI 13-53%, against a pre-registered 45% bar. The control pairs put the judge's same-model noise at a 0.10 deviation from 50%; the cross-arm deviation was 0.21, twice the noise floor, so the verdict is conclusive as a gate decision (directionally clear, though the CI is wide; the rule is a margin-plus-CI rule, not a significance test).

And one number that cuts the other way: citation support. Of the challenger's citations, 51% were supported by the cited page under strict entailment. The incumbent: 33%. The small model's citations, when they existed, verified better.

Finding 1: the failure mode is omission, not fabrication

Every fabrication detector came back clean. Zero invented domains from either model across 116 runs. All trap reports honestly reported scarcity instead of inventing entities. The 2.6B does not make things up at a detectable rate, and neither does the incumbent.

What the 2.6B does instead: 20 of its 58 reports contained zero source URLs. Well-formatted, confidently structured, markdown-clean, and unfalsifiable. The failure concentrates on the harder, sparser queries, and it pairs with length: 473 median words against the incumbent's 189. More words, fewer sources. On one trap query the incumbent cited four real URLs both times; the challenger produced 400-word uncited essays twice.

I find this genuinely more instructive than a hallucination result would have been. Fabrication is the failure everyone tests for, and it did not happen. Omission is quieter. An uncited report does not trip a fact-checker; there is nothing to check. If your acceptance criteria only count invented facts, a model can pass while sliding into prose you cannot verify at all.

The eval literature does treat these as distinct failures: ALCE scores citation recall (uncited claims) separately from citation precision (bad citations), and recent work like Cited but Not Verified measures research agents along exactly this split. What I had not seen stated is how lopsided the split gets at small scale under real agent load: all of this challenger's protocol failure was recall, none of it was precision. There is also a plausible mechanism in the literature: LLMs Get Lost in Multi-Turn Conversation measured an average 39% drop from single-turn to multi-turn performance and attributed most of it to unreliability rather than aptitude. A research agent loop is the multi-turn case by construction. The rule held on turn one and eroded by turn twelve.

Finding 2: decode speed bought nothing

The entire premise of the swap was that a small model makes research faster and lighter. The challenger decodes at ~71 tok/s single-stream on this box; the incumbent is a 26B-class MoE. End-to-end, across 116 runs: 104 seconds mean for the challenger, 107 for the incumbent. The share of runs blowing the 120-second production timeout: 31% versus 32%.

Wall clock in this role is web-search round-trips, page fetches, and tool-call turns. The model's decode speed is a rounding error on top. Regular readers will recognize the shape: this blog spent the summer measuring decode physics, and twice concluded that speed was not the blocker (memory was). Here is the third variant: in an interactive research role, the network is the clock. The tok/s column, the thing the whole small-model case was built on, turned out not to matter at all.

Finding 3: the capability exists, the discipline doesn't

The 51% versus 33% citation-support result deserves its own paragraph, because it complicates the clean story. When the 2.6B cited, it cited more accurately than the model ten times its size. (Both numbers are conservative floors; the entailment check reads raw HTML text windows, and extraction noise hits both arms symmetrically.) The capability to ground claims in sources is present at 2.6B. What is missing is the discipline to keep doing it under multi-step agent pressure, on hard queries, at the end of a long tool loop, with this prompt. That distinction matters for anyone trying to run small models in agent roles: the fix space is prompt hardening and protocol enforcement, not necessarily more parameters. Whether hardening closes the gap is testable, and the bench now exists to test it.

The incumbent, for the record, is not spotless: 81% compliance against its own 95% gate (seven uncited reports, four answers in Finnish despite an English-only contract, on Finland-related topics; Gemma 4 ships with 35+ languages out of the box, and community reports of wrong-language replies from local deployments predate mine), 33% citation support, and the run's only harness timeout. This is the model at the top of a grounding leaderboard, measured in a live role, and the two facts coexist just fine: FACTS Grounding measures faithfulness to provided context in one turn, not citation discipline at the end of a live tool loop. Retention is not an endorsement. It is the pre-registered rule doing its job.

The war story: a judge that measured nothing

The first smoke run produced numbers that looked plausible and were garbage. gpt-oss emits its output in OpenAI's harmony format: an analysis channel (the reasoning) first, then a final channel carrying the actual answer, each wrapped in special-token markup. The final channel arrives last, so a small answer budget gets spent entirely inside the analysis channel and the answer never arrives; OpenAI staff have confirmed exactly this behavior on the model's Hugging Face discussions. My judge harness gave it a 10-token budget. Every response truncated inside the reasoning. The parser then read the truncated markup, found no verdict, and silently defaulted: every judgment parsed as a tie, every entailment as unsupported. Both judging passes truncated identically, so position-consistency read as a perfect 100%.

Ties everywhere, perfect consistency, 0% citation support for both arms. Aggregate metrics that pass a glance and measure nothing. What caught it was not a metric but a smell: a 0% support rate for both arms was implausible, and eyeballing the raw judge outputs showed truncated reasoning markup where verdicts should have been. The fix was a proper final-channel extractor, a 4096-token judge budget, and a rule that truncation now yields "unparseable" rather than a fake verdict.

If you run LLM judges: read the raw outputs before you trust the aggregates, and never let a parse failure default to a valid-looking verdict. My judge had 100% position-consistency while measuring nothing.

What the traps taught

The three fabrication traps were the part of the design I was most curious about. When I reviewed the Feynman research agent, the feature I praised was the verifier that kills dead links and hallucinated references; traps are the adversarial version of the same instinct, aimed at the model instead of the output. They returned a null result that I trust more than a positive one: no invented entities from either model, in any trap report. Both models honestly reported that reliable sources did not exist. Both also got dinged for it, because the compliance rule requires at least one citation per report, and an honest "there is nothing to cite" contains none. The penalty was symmetric, so it did not tilt the verdict, but it is a real flaw in the rule: a compliance metric that cannot distinguish honest abstention from lazy omission penalizes exactly the behavior the traps exist to reward. The next revision of the bench needs an abstention-aware compliance path.

Takeaways

• Bench the role, not the model. The challenger's spec sheet and its mechanical production record both said yes. The role said no. Nothing on a model card measures citation discipline at the end of a twelve-step tool loop on a sparse query.
• A day of rigor beats a month of vibes. The whole thing (literature pass, adversarial methodology review, ~1,100 lines with tests, execution, judgment) was designed, built, and run in one day, and it converted an unfalsifiable hunch into a decision with artifacts. The bench is now a permanent one-command tool; any future small model gets the same audition. I could not find a published example of a home-lab controlled A/B of local models in a live agent role with pre-registered decision rules, which I take not as novelty but as a sign the practice is underused: none of the individual techniques here is original, they are all lifted straight from the eval literature.
• Watch for omission, not just fabrication. At least at this scale and with this prompt, the failure mode of a disciplined small model is not lying. It is confident, well-structured, uncited prose.
• Small-model speed is workload-relative. 71 tok/s of decode bought three seconds of end-to-end latency. In tool-loop roles, the network is the clock. The case for a pinned small model here was never speed; it is residency, and that case survives, waiting on a model (or a prompt) that can keep citing under pressure.

The Taalas post argued that the subagent tier of an agent stack wants a frozen, sufficient, high-volume workhorse, and that small models would increasingly hold those slots. NVIDIA published a whole position paper arguing small models are the future of agentic AI. I still think that is right. This bench adds the qualifier that "sufficient" has to be measured in the role, under load, against a rule you wrote down before you saw the data. My 2.6B candidate was not sufficient yet. The audition process that established that is now the most reusable thing on the box.