Soo... I asked two models the same question through the same harness: "Please explain the silent running technologies in modern submarines. Executive summary only."

The big model made zero tool calls and answered correctly from its own weights: pump-jets, skewed propellers, anechoic tiles, raft-mounted vibration isolation, AIP, the speed trade-off. It even honoured "executive summary only".

The small model made 8 tool calls - 4 web searches, 4 fetches - and led its answer with a submarine propulsion system that does not exist. Not "got a detail wrong": its headline technology was an April Fool's joke, reported as fact.

That result is the post. Retrieval doesn't judge what it retrieves.

The setup

Local: LFM2.5-2.6B at QAD-Q4_0, a 1.48 GB file, llama.cpp on an Apple M5 MacBook with 24 GB, DSpark speculative drafter on top. Measured: 103-113 tok/s decode, ~1,580 tok/s prefill, draft acceptance 68-90%. A laptop, running a genuinely quick small model.

Hosted: Qwen3.8-27B-INT4 on the Trail Openers H100 endpoint, the production workhorse.

Both ran through the same pi harness with byte-identical system prompts (verified by diff) and exactly one tool: a websearch command, ~0.8 s per call. Any difference is the model, not the scaffolding. And to be clear: this is not a fair fight and was never meant to be, one model is ~10x the parameters of the other. The interesting question isn't which wins. It's where the small one breaks.

The anatomy of the submarine answer

Worth walking through, because every failure in it is a failure of judgement, not of mechanics:

  • The lead item came from a page titled "U.S. Navy Submarine First In World Fitted With Silent Caterpillar Drive", dated 1 April 2024, carrying an explicit editor's note: "Important update on this article: April Fool's." The model found it, kept it through six more tool calls, and promoted it to the top of the answer.
  • The technology is fiction two layers deep. The caterpillar drive is from The Hunt for Red October, and the magnetohydrodynamic version isn't even Clancy's: the 1984 novel had a pump-jet tunnel drive, the 1990 film changed it to MHD. A movie's rewrite of a novel's invention, laundered through a prank, into an executive summary!
  • It cited two sources it never loaded. One curl died on DNS (30 seconds burned), the other returned nothing. Cited anyway. The one URL it did fetch returned 50 KB of WordPress cache-plugin HTML.
  • It inflated the one real thing it found: HRL's 2026 MHD concept is genuine research, and the PI's own words are "we hope to next have the opportunity to build a complete prototype." The model rendered that as the U.S. Navy having fitted its first submarine.
  • And it omitted what is actually on every modern boat, pump-jets and anechoic tiles, which the zero-search model led with.

n=1, single trial, disclosed as such. But the pattern it illustrates reproduced everywhere else in the session.

Where the risk actually peaks

Here's the counterintuitive part, and I'd say it's the most useful thing in the whole exercise: hallucination risk peaks at partial knowledge, not zero knowledge.

On a question where the model has essentially no training data (a niche Finnish topic), it cleanly said "I don't know". Genuinely good behaviour for 2.6B: it also held that line under social pressure, and separately refused to cave to a false correction, calling it "a common misconception". Correctly!

But ask near the edge of what it knows and the same model recombines fragments with full confidence. Asked about the licorice company in Vaajakoski, it correctly declined to invent a company name... and then volunteered, unprompted, that Vaajakoski is a small town in Päijät-Häme near Tampere. Three errors in one clause: it's a district of Jyväskylä, in Central Finland, nowhere near Tampere. Asked again, 3 runs out of 3 placed it in "Kuhmo municipality, part of the Päijät-Häme region", which is internally impossible - Kuhmo is in Kainuu, hundreds of kilometres away. It also suggested searching for licorice with the term "Suklaa". That's chocolate.

Careful about what it was asked. Careless about what it volunteered. The disclaimer earns trust that the adjacent sentence then spends. And the failure curve explains the submarine too: submarines sit exactly in the partial-knowledge band, enough fragments to feel confident, not enough to notice that a caterpillar drive belongs to Sean Connery.

Tools fix the easy case, to be fair: given the websearch tool and the single-fact question, it got Vaajakoski exactly right, sources named, in 11 seconds. Retrieval genuinely repairs single-fact grounding. It just cannot repair judgement, because judgement is what decides which retrieved page is a prank.

The agent question, honestly

In this laptop setup, compound questions broke it: "Where is Vaajakoski?" works in 11 seconds, add a second clause and it loops until killed, 3 out of 3, and a prompt-level guard ("at most two searches, never repeat one") did not hold. Qwen took the same compound question in 16 seconds.

Now, I run this exact model as the research assistant inside my personal assistant stack, where it has done multi-day production stints: 1.46 million prompt tokens, zero tool-call parse failures, zero malformed JSON. So I'd put it this way: the looping is setup-sensitive (prompt sensitivity was real throughout this session, and some of what looks like a model limit may be promptable). The knowledge poverty is the constant. It showed up in the controlled bench as citation discipline collapsing on hard queries, and it shows up here as an April Fool's page winning the source-selection contest. Same root, two costumes.

What the 1.48 GB file is actually for

This isn't "small model bad". It declined to invent a company, resisted a false correction, summarises beautifully, and runs at 113 tok/s on a laptop while drawing basically nothing. It's a good reader and a capable executor. It is a very poor knower - and it cannot tell you which of those modes it's in.

So the practical advice writes itself: supply the facts, keep the grounding contract tight, and never let it decide alone what's true from what it finds. Source judgement has to live outside the model: in allowlists, in verification steps, in a bigger model reviewing, in you.

The part I keep chewing on: we measure models on what they know and what they can fetch. The submarine test says the scarcer skill is knowing when a fetched thing should be thrown back...