Not everything investigated for halo gets adopted, and the reasons why are often more useful than another adoption story. Three items from the recent research pile: two dead ends worth knowing about, and one unfreeze that was pure win.

DwarfStar: real support, wrong backend

DwarfStar (antirez/ds4) is Salvatore Sanfilippo's local inference engine, 22,000+ stars, built primarily for the DeepSeek V4 family. It came onto the radar because of a recollection of impressive Qwen3.8-Flash-Next numbers floating around - and the support turned out to be real: a merged PR adding native qwen4exp support, benchmarks through 262K tokens, actively maintained.

Two catches. First, the backend: at investigation time the model's dedicated kernel graph was documented as Metal-only, and while the engine has since broadened (it now targets Metal, CUDA and ROCm generically, with the Flash-Next path on Metal and CUDA), there is still no ROCm implementation of this model's kernels - and halo is a ROCm/Vulkan box. Second, the packaging: DwarfStar's Flash-Next builds run 137-165 GiB on disk with a 95 GB n-gram table read straight from the GGUF, a fundamentally heavier shape than our ~35 GB quant plus a 4 GB sidecar. A well-built project that simply is not for this hardware yet. Worth a re-check when the backend matrix grows; not before.

The meta-lesson from how this was investigated: the first research pass reported the ds4 PR as "still open, weak numbers". An independent gh pr view showed it had merged the day before, with materially different numbers than reported. Never trust a single research pass on anything that will inform a real decision - a rule this blog has now paid for and been paid by several times.

DiffusionGemma: served by vLLM, not by llama.cpp

DiffusionGemma is Google's experimental text-diffusion take on Gemma-4: the same 25.2B/3.8B-active MoE backbone, fine-tuned into a block-diffusion model. Instead of predicting one token at a time, a causal encoder reads the prompt and a bidirectional decoder iteratively refines a 256-token block, roughly 12 forward passes per block and ~20 tokens per pass - Google cites ~1,500 tok/s on a single H100. Genuinely different decoding economics, which is exactly why it's interesting for this hardware class.

The catch for halo is where the support landed. vLLM serves it natively. llama.cpp does not: support lives in an unmerged draft PR that adds a dedicated llama-diffusion-cli binary, and llama-server - the OpenAI-compatible path this whole box's architecture depends on - fails on the model with an unknown-architecture error on both ROCm and Vulkan. There's even a public failure report from another Strix Halo box hitting exactly that, and the community workaround is a Python proxy that shells out to the CLI. A one-off CLI behind a proxy is a demo, not a route.

So the accurate status: the serving path exists, it just lives in vLLM, and halo's entire stack is llama.cpp. Filed under revisit-when-llama-server-support-lands, which I'd say is the correct amount of enthusiasm for a model this box can build but not serve.

The ROCm 10 unfreeze

The quiet win of the batch. Halo's production ROCm container had been frozen at a six-week-old llama.cpp build since early August, pinned there to dodge a HIP integrated-GPU corruption bug (#25992) whose complete fix never merged. Then kyuz0's toolboxes moved to ROCm 10, where gfx1151 graduates from tolerated to an AMD-packaged, officially supported target, with the workaround baked in.

Evaluated with the usual paranoia before touching production: the corruption symptom exercised directly (8/8 concurrent distinct prompts, no cross-contamination), then a speed A/B against a freshly-taken baseline on the frozen build, not stale historical numbers. Result: prefill +10.8% at 8K depth and +17.5% at 16K, decode flat. For a box whose stated priority is prefill at depth for coding agents, that's the right win in the right place, for free. Adopted; the frozen container stays on disk as rollback.

One migration note for anyone following the same path: the new builds have removed --no-mmap outright in favour of --load-mode, which breaks any script that hardcodes the old flag - including, at time of writing, the strix-halo-optimize bench scripts on this class of toolbox. Flag shims needed; the skill's findings log has the details.

Three investigations, one adoption. The ecosystem moves fast enough that the dead ends of September are worth re-checking by November - which is, I suppose, exactly why the journal exists...

The coding-model roster on halo got a proper shootout: four models, two benchmark systems, every number measured on this box. Which model won? That turns out to be the less interesting question. The finding I'd actually lead with: pass@1, the number everyone quotes (the share of problems a model solves on its first try), quietly merges two completely different ways of failing - answering wrong, and never answering at all. And the model ranking changes depending on which failure your workload actually cares about.

The roster

Halo (the AMD Ryzen AI MAX+ 395 box from the setup guide, 128 GB unified memory) currently serves four coding models via llama-swap:

• gemma-4-26b-a4b: Google's Gemma-4 MoE, 25.2B total / 3.8B active, the box's strongest single coding model and the production grounding champion elsewhere in the stack. Now also vision-capable here (more on that another day). A second route runs the same weights with an official Unsloth MTP speculative-decoding sidecar: +68-73% decode, measured.
• qwopus3.6-35b-coder: a community fine-tune of the Qwen3.6-35B-A3B lineage, thinking-off by default, tuned for fast read/edit/test/fix agent loops. The default fan-out model, serving 6 concurrent slots.
• qwen3.8-flash-next-mtp: the Qwen4-preview architecture, now with its native MTP head working via a community-built toolbox. That story earned its own post, coming next.
• north-mini-code: CohereLabs' North-Mini-Code-1.0, a 30B-A3B MoE, smoke-tested this week and not yet in production routing.

Housekeeping note with a number attached: this roster review also decommissioned six models (the whole Nemotron-3 line, Qwen3-Coder-Next, Ornith-1.5, Qwen3.5-122B), freeing 243 GB of disk. Model churn is real; the audition process keeps working in both directions.

Getting comparable numbers: EvalPlus, and the bug that nearly poisoned them

Our own to-bench-style harnesses are great for A/B work but mean nothing to anyone else's hardware. So the roster got scored on EvalPlus: HumanEval+ and MBPP+, the standard, globally-comparable Python coding benchmarks. Anyone can put these numbers next to published ones, which is the whole point of adopting them.

Except the first numbers were garbage, and the way they were garbage is the instructive part. EvalPlus's OpenAI-API backend hardcodes a 768-token generation cap, no flag to raise it. Reasoning models spend their token budget thinking before any code appears - so the harness was silently truncating them mid-thought and scoring the empty result as a wrong answer. The tell was an anomalous number of empty completions. Raising the cap to 8192 (a monkeypatch; there's no flag) fixed most of it, and then revealed a second, genuine finding: some completions stay empty at any budget. On a reproducible fraction of problems, these models reason indefinitely and never answer. Same problems, every run, multiple budgets: a characteristic, not an artifact.

The per-model empty rates at 8192 tokens: qwen3.8-flash-next 12/164 on HumanEval+ and 27/378 on MBPP+ (~7%), north-mini-code 5/164 and 9/378 (~2%), qwopus 1/378, and gemma-4 zero throughout - not because it never overthinks, but because its production config caps reasoning at 1024 tokens, which prevents the failure by construction. Empties are counted as fails in the scores below, which I'd argue is correct: an agent that never answers has failed the task either way.

One more trap for the pile: EvalPlus silently skips re-scoring when a results file already exists. After the patch, the scores did not change until a --i_just_wanna_run flag forced re-evaluation. The series tradition of measuring the wrong thing without an error message continues: your benchmark harness is part of the experiment.

The scores

pass@1, greedy decoding, temperature 0:

Read column by column and the tidy ranking dissolves. HumanEval+ shows Gemma-4 clearly ahead. MBPP+, with shorter and more constrained problems, narrows everything and flips the base ranking outright: North Mini's 0.937 against Gemma-4's 0.939 is a statistical tie at n=378. And the MBPP+ plus column is led by qwen3.8-flash-next, the model with the worst rate of never-finishes-thinking failures in the whole roster.

That last one is the takeaway I'd underline. Flash-Next's failures cluster almost entirely in "never answers", not "answers wrong": when it produces code, the code tends to be correct. Think about what that means for model selection: a model that answers everything with 92% accuracy and a model that answers 92% of problems with near-perfect accuracy carry the same pass@1, and they are completely different tools. One needs review. The other needs a timeout. Raw pass@1 tells you neither, and I'd guess most model-picking decisions out there never split the two.

Honest limits, stated up front rather than in a footnote: EvalPlus is Python-only, single-turn, zero tool use. It does not test the agentic terminal work these models actually do all day. Portable signal, not the whole picture - and the agentic-benchmark gap is now the most obvious hole in this box's evaluation suite.

The realistic speed sweep

For speed, the usual llama-bench synthetic-token corpus was deliberately rejected. Prefill content: an ~855 KB concatenation of the transformers library's actual modeling_*.py source files, sliced at three depths, with prompt caching forced off so repeated depths get no cache discount. Decode task: implement a thread-safe LRU cache with tests. Numbers read from the server's own reported timing fields, not wall-clock. And one deliberate choice worth naming: each model ran its own production route (its real backend, quant, and flags), because the question was "how fast is this model as I actually serve it", not a backend-controlled lab comparison.

Caveat first, because it's a real one: three of the four decode numbers measure reasoning-token throughput, not code-token throughput - the sweep didn't force thinking off, and only Gemma-4's 1024-token reasoning cap left room for actual code inside the test budget. Architecturally, next-token decode cost doesn't depend on what the tokens say, so it's very likely a fair proxy. Flagged, not buried; re-running with thinking forced off is on the list.

The structural finding survives any caveat: Flash-Next's prefill sits at a third to a quarter of the other three, and no amount of MTP decode speedup touches that. Agentic work is prefill-dominated - agents re-read code and re-send long context constantly - so this is the bottleneck that decides real workloads. The receipts are in the benchmark wall-clocks themselves: Flash-Next took ~3.5 hours for HumanEval+ against ~1.5 for the others, and ~8.5 hours for MBPP+ against ~2.5-3, while scoring similar-or-better per problem. Good answers, slowly: the day-one assessment of an immature kernel stack still stands.

Where this leaves the roster

Gemma-4 keeps the crown for single-shot quality, and its MTP route makes it quick too. The fan-out slots stay with qwopus3.6: 64.5 t/s decode, competitive prefill, thinking-off, six parallel slots - the shape agent orchestration wants. North Mini Code earned a real decision: statistically tied with the champion on MBPP+, fastest decode on the box, and not yet wired into production. Whether it displaces anything is exactly what the missing agentic benchmark should decide, and building that benchmark is now the top open thread.

So the roster verdict is provisional in a specific, stated way: single-turn Python is measured, the day job isn't yet. And the question a year into this series keeps getting less like "which model is best" and more like "best at which failure mode, at which speed, in which slot". Early days on that agentic benchmark tho...

MTP - multi-token prediction, the speculative decoding where a model drafts its own future tokens with a built-in head instead of a separate draft model - has had a rough arc on this blog: a tuning win that required rejecting the community default, a measured quality cost on hard tasks, a cross-engine divergence confirmation at temperature 0. The running advice was: measure before you trust it.

Soo... we measured. A lot. The result is that halo now runs two MTP routes in production: Gemma-4-26B at +69% decode, and Qwen3.8-Flash-Next at a campaign-validated +51% mean. This post is how they earned their way in, including the backend that got banned and the claim of ours that did not survive its own validation campaign.

Gemma-4: the easy one

Unsloth publishes an official MTP draft sidecar for Gemma-4-26B-A4B: a 441 MB "smart Q4_0" drafter that llama.cpp has supported since June. Wire it in with --spec-draft-model and --spec-type draft-mtp, and on this box: 40.88 to 68.97 t/s on a short real prompt (+68.7%, 78.7% acceptance), 70.68 t/s sustained through a 2500-token generation at 82% acceptance, no collapse, no corruption.

Correctness got the treatment the quality-cost findings made mandatory: temp-0 comparison (not byte-identical, as expected by now - more below), then the generated code extracted and executed against real test cases. Passed.

One deployment decision worth copying: the MTP route is a separate, opt-in, single-slot route (gemma-4-26b-a4b-mtp), not a replacement for the production multi-slot route. Why? MTP gains invert under concurrency on this hardware - measured for the Qwen3.6 family, inherited here as the safe default rather than re-proven for Gemma-4 specifically. Batching already amortises what speculation buys; the H100 team found the same shape independently. Single-stream lever, single-stream route.

Flash-Next: the hard one

Qwen3.8-Flash-Next ships a native MTP head, but llama.cpp support for it still lives in open PRs (#27836 base support, unmerged as of this writing). The working path came from the community: a member (drluoto) assembled a branch combining the PR head with the loader and tensor-naming fixes needed to actually run it, and that branch became a custom ROCm toolbox here. Our base quant (UD-IQ3_XXS) paired with the Q8_0 sidecar was a combination nobody had published numbers for.

The first smoke test looked almost too good: 24.84 to 45.30 t/s, +82%, 97.2% acceptance. House rules say a single prompt adopts nothing, so it got a campaign: 5 real to-bench task prompts × 3 seeds, MTP-on vs MTP-off, paired. Result: +51.1% mean speedup (range +36.2% to +60.7%, stdev 8.2%), acceptance 87.4%, consistent across every pair. Then a third, independent cross-check on fresh real prompts that were never part of the campaign: +38.5% and +57.0%, landing inside the campaign's range. Three methods, one effect size. That's an adoption.

And a swap worth a paragraph: when Unsloth later published an official sidecar for this model, it got A/B'd against drluoto's on identical prompts. 40.55 vs 39.59 t/s, 90.5% vs 89.5% acceptance - essentially equivalent, slight edge to official. Adopted anyway, for the boring reason: a maintained official source beats a community rehost that could go stale. The community sidecar stays on disk as rollback. Community work got us here months early; official sources are what you settle on.

The backend that got banned

This box's default backend preference is Vulkan RADV for most things. For MTP it inverted, and the way it inverted is the cautionary tale.

On a short 143-token prompt, Vulkan MTP looked like a clean win: +60%, 94.4% acceptance. On a longer, harder 2000-token generation, throughput collapsed to a steady 6.3-6.6 t/s - not a crash, a collapse, sitting at a fifth of the no-speculation baseline (29.33 t/s) from early in the generation to the end. A short-prompt-only test would have shipped a severe regression into production with a green checkmark on it.

Honest scoping: the short and long tests differed in context size and prompt content, not just length, so the exact trigger wasn't isolated. But the operational conclusion needed no more precision: ROCm is the only backend validated safe for MTP on this hardware, through full-length generations. Benchmark depth is not optional. It's where this class of failure lives.

The claim we had to correct

That first Flash-Next smoke test also reported MTP-on and MTP-off producing byte-identical output at temperature 0. The validation campaign proved our own claim wrong: 0 of 15 pairs were byte-identical. Small floating-point-driven divergence appeared in every single pair; the smoke-test prompt just happened not to trigger it.

What was preserved is the thing that matters: functional correctness. Five pairs hit token-budget truncation in both arms and weren't comparable; of the comparable pairs, all but one had identical pytest pass/fail outcomes on the extracted code, and the one divergent pair scored better with MTP on (12/13 vs 10/13) - verified by full pytest diff to be a genuinely different-but-valid implementation, not corruption.

This slots cleanly into the two-mechanisms picture this blog ended up with: divergence is universal (the losslessness proof assumes exact arithmetic, hardware disagrees), and whether it costs anything is a per-model, per-head question only an executed benchmark can answer. Ornith's immature head cost 17-25% on hard tasks. Flash-Next's, on this evidence, costs nothing measurable. Both posts now cite each other, because readers of either need the other half.

Where this lands

I'd put the state of MTP on this hardware like this: it is neither the free speedup the leaderboards imply nor the quality hazard a divergence report implies. It is a per-model, per-backend, per-workload lever that pays +40-70% single-stream when the head is well-trained, inverts under concurrency, collapses on the wrong backend, and always changes outputs at the bit level without necessarily changing them at the level that matters. Every clause in that sentence was measured on this box, most of them twice.

The validation tax for the two adoptions was real: campaigns, cross-checks, temp-0 diffs, executed code. Also, I'd say, obviously worth it: the alternative was either leaving 50-70% of decode on the table or shipping a Vulkan collapse. Measure before you trust it still stands. It's just that now, twice, the measurement said yes...

Every speed number this blog has published for halo rests on llama-bench, and llama-bench's prompts are synthetic: repeated tokens, not real text. A reader raised the reasonable question - do those figures reflect real usage at all, or are they an artifact of the corpus? Fair challenge. The answer to a fair challenge is a measurement, not a defence.

The cross-check: real prompt content (concatenated coding-task prompts, reference solutions, and a genuine coding question) at three sizes - 206, 3,192, and 6,421 real tokens - run three times each on Qwen3.8-Flash-Next UD-IQ3_XXS, on both ROCm and Vulkan backends, and compared against the synthetic llama-bench sweeps at the corresponding depths. Decode measured the same way: 250 real generated tokens on the same prompts.

The result, backend by backend

ROCm: essentially an exact match. The real 3,192-token prompt prefilled at 392.98 t/s. The synthetic pp4096 sweep: 390.78 t/s. That's 0.6% apart, within run-to-run noise. The 6,421-token real prompt (371.10 t/s) fits smoothly on the synthetic declining-with-depth curve. Decode: real-prompt numbers land 4.9% from the synthetic figure at comparable depth, with rep-to-rep spread of 0.05-0.21 t/s. Hard to ask for better.

Vulkan: a real 8.7% gap, disclosed rather than papered over. Real 3,192-token prefill measured 344.09 t/s against synthetic pp4096's 377.07. The plausible explanation: llama-bench's chunked-at-depth measurement and a fresh single-shot prompt of comparable length are not strictly identical measurements - attention cost accumulates differently - so the gap doesn't necessarily mean either number is wrong. But it's real, it's backend-specific, and if you quote Vulkan synthetic numbers, real prompts on this box run a bit slower than they suggest.

One number that looks alarming and isn't: the short 206-token real prompt measured far below the synthetic short-depth figure (182.63 vs 431-ish on ROCm). Not a discrepancy - llama-bench's "pp0" measures a 2,048-token chunk, and small-batch fixed overhead dominates a 206-token prompt. Not a like-for-like comparison, so don't read it as one.

Worth saying why this check mattered extra on this particular model: it's a MoE, and expert routing is content-dependent. Synthetic repeated tokens could, in principle, exercise a completely different expert-load pattern than real code. Measured verdict: they don't, at least not enough to matter on ROCm.

The bug the check itself caught

The first attempt at this cross-check produced garbage, and the way it did is the transferable lesson. Reps 2 and 3 of each prompt came back suspiciously fast - because llama-server's prompt cache silently recognised the repeated prompt and collapsed the prefill to a handful of tokens. The "benchmark" was measuring cache lookups. Caught by inspecting the per-rep prompt_n fields, fixed with cache_prompt:false, whole exercise redone.

That's the third measured-the-wrong-thing incident this series (the judge, the stale server, now the cache), and I'd say the pattern is now a rule: any live-serving benchmark must either disable prompt caching or vary the prompt per rep, and you verify which happened from the server's own counters, not from the wall clock looking plausible.

Where this leaves the numbers

So: synthetic benchmarks, on this box, on this model class, measure the real thing - on ROCm nearly exactly, on Vulkan with a known, quantified offset. The MTP campaign numbers got the same treatment (fresh real prompts, never used in the original campaign) and converged on the same effect size, which is what let those adoptions ship with confidence.

The general habit I'd recommend from all this: don't defend your benchmarks, cross-check them. It cost one afternoon, it answered the skeptic properly, and it found a methodology bug that would have quietly poisoned some future measurement anyway...

Well.. here's a result that sounds wrong until you see the mechanism: Cerebras runs Qwen3.8-27B about eleven times faster than our H100, and still finishes fewer agent turns per minute than that H100. Both halves measured, same day, same harness, same model on both sides.

Let me be precise about what this is not saying. Cerebras is not slow. Single stream it decoded at 728-1142 tok/s in our tests against our box's ~87. The wafer-scale silicon delivers exactly what the marketing says, and if your workload is one stream of raw generation, it wins by a mile and this post does not apply to you. What follows is about a different question: what happens when you point a real agent harness at it.

The setup

Same model on both sides: Qwen3.8-27B, Cerebras' hosted qwen-3.8-27b versus RedHatAI's INT4 on the Trail Openers H100 in Helsinki. Both driven through the same OpenAI-compatible harness (pi-rukas, the agent framework we use daily): identical prompts, identical token budgets, reasoning_effort: "none" on both so neither burns thinking tokens, prefix caching warm on both, 429s retried with backoff rather than counted as failures. Six tests, about 22 minutes under load, 16.9.2026.

Agent work has a particular shape, and it decides everything here: the harness resends the whole conversation every turn. A coding agent sitting on a ~38k-token repo context sends those 38k tokens again and again, ~97% of them cache hits, to generate a few hundred new ones each time.

What the six tests said

Raw speed: Cerebras, decisively. 988 tok/s median decode vs 87. No contest, no dispute. (Funny detail: our time-to-first-token is better, 0.14s vs 0.33s, because the box is close and uncontended. They win the moment generation starts, not before it.)

Warm cached turns: us, actually. A 30k-token system prompt, four sequential turns, both sides caching. Their warm turns: 0.49-0.67s. Ours: 0.22-0.28s. For a short answer over a cached context - the single most common agent turn there is - our endpoint is already faster end-to-end.

Concurrency: we scale, they flatline. Simulated agent sessions over a ~38k context: Cerebras sits at 15-16 turns/min whether you run 1, 4, or 12 parallel sessions. Twelve sessions produced 60 rate-limit rejections and zero extra work. Our box went 9.2 → 27.5 → 45.6 → 62.7 turns/min at 1/4/12/24 sessions, zero rejections, still climbing when I stopped.

The obvious rebuttal, controlled for. Cerebras reserves input + max_completion_tokens up front, so surely the harness's 32,768-token default is self-inflicted? Re-ran the whole sweep at max_tokens: 1024, which is their own documented fix for spurious 429s. It changed nothing: 13.9-16.9 turns/min across the board.

The crossover: ~13k tokens of context. At ~10k context Cerebras wins clearly (62.8 vs 32.2 turns/min). At ~13k it's a dead heat. At ~38k we do 27.6 to their 12.4, and our box pushed 1.08M total tokens/min while theirs pinned at 460-670k no matter how the work was shaped.

What it feels like in practice: blocked. With the harness honouring their real Retry-After: 59, four sessions at 38.6k context spent 714 of 768 session-seconds waiting on backoff. 93% of wall-clock, blocked. The "it feels slow" instinct was correct: it was watching a full-minute stall, several times per session.

The mechanism, because it predicts every number above

Cerebras enforces two per-minute token buckets: an uncached one (150,000 tok/min on this account) and a total one at three times that, ~450k. Cache hits are exempt from the uncached bucket. They are not exempt from the total bucket. So a 38k-token agent context costs 38k against the quota every single turn, even when 97% of it is a cache hit and the model generates 300 new tokens. Our measured uncached burn was 19,248 tok/min, 13% of the allowance - we were nowhere near the advertised limit and throttled anyway.

Which gives you a rule simple enough to say out loud: turns per minute ≈ the total-token ceiling divided by your context size. Below ~13k of context the fast silicon wins. Above it, the ceiling does the deciding, and the world's fastest inference sits idle waiting for the next minute to start.

What I'd actually claim

So I'd put it this way: this is not a hardware story, it's a quota-design story. Their limiter prices a cached token like a computed one, and agents are the workload that maximises exactly that ratio. The unarguable version of the finding: dedicated capacity beats metered capacity for sustained agentic work. Our H100 has no rate limits because it's ours; that, not the silicon, is what won.

The caveats, before someone else states them: this is one account on one tier (450 RPM / 150k uncached TPM, measured 16.9.2026), not "Cerebras" - enterprise and dedicated endpoints get provisioned differently. Theirs is a shared multi-tenant service measured at one point in time. No cost claim here: a metered API against a box rented 24/7 is not a like-for-like bill. And no quality claim: they serve unquantised weights, we serve INT4; throughput compared, output quality not.

The part I find genuinely interesting: every provider racing to the top of the tok/s leaderboards is optimising the number that matters least for agents. An agent's real currency is completed turns per minute at its actual context size, and that's decided by quota shape, cache pricing, and backoff behaviour - the things no leaderboard shows. Speed-of-light decode with a context-priced meter in front of it is a sports car with a toll booth every 38,000 tokens...

Yesterday I gave a talk at a client about running language models on your own hardware. The slides that drew the most questions were not about capability. They were about what one ordinary afternoon of work actually costs, so here they are as a post.

The afternoon in question: Friday 2026-09-11, the production H100 we run at Trail Openers in UpCloud's Helsinki data centre, Qwen3.8-27B-INT4 under vLLM, serving agent-driven coding work. Not a demo, a normal workday. Seven hours: 1.10 billion input tokens presented to the model, 3.24 million generated, 3.77 kWh, prefix-cache hit rate 97.5%. Cost: 12,53 €, because the machine rents at 1,79 €/h and a fixed-price card costs the same busy or idle.

Two numbers in that paragraph explain everything that follows. Agentic coding is input-heavy in a ratio of roughly 380:1 - the agent reads a lot of code and writes a small diff. And of that billion-token firehose, only 27.5 million tokens were actually computed. The rest came from the prefix cache: the same conversation history, resent every turn, recognised instead of recomputed.

The same afternoon, bought from an API

Take the exact same token volumes (27.5M new input, 1 077M cache reads, 3.24M output) and price them at published rates (read 13.9.2026, 1 € = 1,09 $):

The comparison people expect is the first row against the last: 58x. The comparison worth learning from is the middle of the table. Darkbloom has the cheapest input price on the list and no cache pricing; Chutes charges twice the list price and offers cache reads at 0,032 $. On this workload the "cheap" provider costs over three times more. The bill is decided by the price of a cache-read token, not by the input list price. Once you know the workload is 380:1 input-heavy and nearly all of that input repeats, the pricing page reads completely differently: the cache-read column is the price, everything else is footnotes.

The honest caveats, because they were on the slides too

The 97.5% hit rate is measured on our vLLM endpoint and assumed for the API columns. Anthropic's cache expires in five minutes and behaves differently; nobody has measured our workload against it. The own-machine figure excludes the maintenance work, which is real. And one more, the one I insist on saying out loud: our card was cheapest because it is mostly idle on our workload shape, median concurrency 0.6 over 21 production days. Per actually computed token our cost is about 0,13 €/M, which is list-price class - a provider who keeps their GPUs full produces a token cheaper than one card ever will. The fixed 1,79 €/h is not magic efficiency. It is a fixed cost meeting a cache-friendly workload.

Over those 21 days the picture holds at scale: 26.4 billion tokens processed for 866 € of machine time, against roughly 16 700 € priced on Claude Sonnet 5 rates. Same mechanism, bigger window.

So I'd put it this way: the per-token price war is real, and on raw computed tokens the APIs are competitive with a self-hosted card. What you are actually buying with your own machine is a different thing: a fixed cost ceiling, EU data residency, and a cache that belongs to you and never expires on a five-minute timer. Whether that is worth the ops work depends entirely on how your workload shapes up.

Mine, apparently, shapes up to 12,53 € an afternoon. The talk's capability and energy slides might be worth their own posts too...

Soo... I asked two models the same question through the same harness: "Please explain the silent running technologies in modern submarines. Executive summary only."

The big model made zero tool calls and answered correctly from its own weights: pump-jets, skewed propellers, anechoic tiles, raft-mounted vibration isolation, AIP, the speed trade-off. It even honoured "executive summary only".

The small model made 8 tool calls - 4 web searches, 4 fetches - and led its answer with a submarine propulsion system that does not exist. Not "got a detail wrong": its headline technology was an April Fool's joke, reported as fact.

That result is the post. Retrieval doesn't judge what it retrieves.

The setup

Local: LFM2.5-2.6B at QAD-Q4_0, a 1.48 GB file, llama.cpp on an Apple M5 MacBook with 24 GB, DSpark speculative drafter on top. Measured: 103-113 tok/s decode, ~1,580 tok/s prefill, draft acceptance 68-90%. A laptop, running a genuinely quick small model.

Hosted: Qwen3.8-27B-INT4 on the Trail Openers H100 endpoint, the production workhorse.

Both ran through the same pi harness with byte-identical system prompts (verified by diff) and exactly one tool: a websearch command, ~0.8 s per call. Any difference is the model, not the scaffolding. And to be clear: this is not a fair fight and was never meant to be, one model is ~10x the parameters of the other. The interesting question isn't which wins. It's where the small one breaks.

The anatomy of the submarine answer

Worth walking through, because every failure in it is a failure of judgement, not of mechanics:

• The lead item came from a page titled "U.S. Navy Submarine First In World Fitted With Silent Caterpillar Drive", dated 1 April 2024, carrying an explicit editor's note: "Important update on this article: April Fool's." The model found it, kept it through six more tool calls, and promoted it to the top of the answer.
• The technology is fiction two layers deep. The caterpillar drive is from The Hunt for Red October, and the magnetohydrodynamic version isn't even Clancy's: the 1984 novel had a pump-jet tunnel drive, the 1990 film changed it to MHD. A movie's rewrite of a novel's invention, laundered through a prank, into an executive summary!
• It cited two sources it never loaded. One curl died on DNS (30 seconds burned), the other returned nothing. Cited anyway. The one URL it did fetch returned 50 KB of WordPress cache-plugin HTML.
• It inflated the one real thing it found: HRL's 2026 MHD concept is genuine research, and the PI's own words are "we hope to next have the opportunity to build a complete prototype." The model rendered that as the U.S. Navy having fitted its first submarine.
• And it omitted what is actually on every modern boat, pump-jets and anechoic tiles, which the zero-search model led with.

n=1, single trial, disclosed as such. But the pattern it illustrates reproduced everywhere else in the session.

Where the risk actually peaks

Here's the counterintuitive part, and I'd say it's the most useful thing in the whole exercise: hallucination risk peaks at partial knowledge, not zero knowledge.

On a question where the model has essentially no training data (a niche Finnish topic), it cleanly said "I don't know". Genuinely good behaviour for 2.6B: it also held that line under social pressure, and separately refused to cave to a false correction, calling it "a common misconception". Correctly!

But ask near the edge of what it knows and the same model recombines fragments with full confidence. Asked about the licorice company in Vaajakoski, it correctly declined to invent a company name... and then volunteered, unprompted, that Vaajakoski is a small town in Päijät-Häme near Tampere. Three errors in one clause: it's a district of Jyväskylä, in Central Finland, nowhere near Tampere. Asked again, 3 runs out of 3 placed it in "Kuhmo municipality, part of the Päijät-Häme region", which is internally impossible - Kuhmo is in Kainuu, hundreds of kilometres away. It also suggested searching for licorice with the term "Suklaa". That's chocolate.

Careful about what it was asked. Careless about what it volunteered. The disclaimer earns trust that the adjacent sentence then spends. And the failure curve explains the submarine too: submarines sit exactly in the partial-knowledge band, enough fragments to feel confident, not enough to notice that a caterpillar drive belongs to Sean Connery.

Tools fix the easy case, to be fair: given the websearch tool and the single-fact question, it got Vaajakoski exactly right, sources named, in 11 seconds. Retrieval genuinely repairs single-fact grounding. It just cannot repair judgement, because judgement is what decides which retrieved page is a prank.

The agent question, honestly

In this laptop setup, compound questions broke it: "Where is Vaajakoski?" works in 11 seconds, add a second clause and it loops until killed, 3 out of 3, and a prompt-level guard ("at most two searches, never repeat one") did not hold. Qwen took the same compound question in 16 seconds.

Now, I run this exact model as the research assistant inside my personal assistant stack, where it has done multi-day production stints: 1.46 million prompt tokens, zero tool-call parse failures, zero malformed JSON. So I'd put it this way: the looping is setup-sensitive (prompt sensitivity was real throughout this session, and some of what looks like a model limit may be promptable). The knowledge poverty is the constant. It showed up in the controlled bench as citation discipline collapsing on hard queries, and it shows up here as an April Fool's page winning the source-selection contest. Same root, two costumes.

What the 1.48 GB file is actually for

This isn't "small model bad". It declined to invent a company, resisted a false correction, summarises beautifully, and runs at 113 tok/s on a laptop while drawing basically nothing. It's a good reader and a capable executor. It is a very poor knower - and it cannot tell you which of those modes it's in.

So the practical advice writes itself: supply the facts, keep the grounding contract tight, and never let it decide alone what's true from what it finds. Source judgement has to live outside the model: in allowlists, in verification steps, in a bigger model reviewing, in you.

The part I keep chewing on: we measure models on what they know and what they can fetch. The submarine test says the scarcer skill is knowing when a fetched thing should be thrown back...

This blog's MTP scoreboard so far: a throughput win on Strix Halo that required rejecting the community default, a measured quality cost on hybrid-GDN models in two engines, and a "don't enable it" verdict on the H100's MoE workload. This post adds the first entry where MTP went into production: the Trail Openers H100 endpoint moved to an INT4 checkpoint of Qwen3.8-27B, and that swap unlocked a usable MTP head. The numbers say run it. The mechanism story says keep watching, and it corrects something this blog previously stated too strongly.

Why the quant swap unlocked MTP: the INT4 source ships its 849 MB MTP head in BF16, explicitly held out of quantization. One trained layer, so draft depth 1 is the only native depth. Regular readers will recognize that as the depth our halo tuning independently converged on.

The number that matters: end-to-end request time

Not decode rate. Full HTTP round-trip from the client, TTFT included, at concurrency 4 with ~36K-token prompts and 145-token completions (145 is production's mean completion length; the median is 20-50 tokens, and 89% of real completions are under 200):

n=10 per arm, mean ratio 1.41x, median 1.30x, t=3.91. The variance drop is arguably the better half of the result: a 3x tighter spread is what an interactive agent user actually feels.

The counterintuitive part worth reporting: TTFT roughly doubled (0.61 s to 1.2 s), and total time still fell. The verification pass makes the first token more expensive; every step after it yields more than one token; over even a short completion the repayment wins. If you monitor TTFT as your latency proxy, MTP will look like a regression while making every request faster.

Caveats, stated plainly: synthetic shared-prefix prompts at 36K (production averages are longer), an otherwise idle box, and one earlier n=1 baseline measurement (2.90 s) that contradicts the n=10 distribution and is disclosed rather than hidden.

Why acceptance is high here, and what it cost

Draft acceptance ran 93-94% at 400-token completions and 82-88% on the short reps. The series now has an acceptance ladder, and it tracks how purpose-built the drafter is:

Costs: the KV pool shrank 9.1% (max concurrency at full context 6.36x to 5.78x), mild next to the 41.6% the DFlash drafter cost. Two sharp edges worth copying into your own checklist: min_p and logit_bias silently stop working under speculative decoding in vLLM (it warns once at boot, then fails quietly), and one known open vLLM spec-decode bug is present in this image version but did not fire in testing.

The divergence, measured

At temperature 0, five coding probes, three repeats per arm: both arms fully self-stable, and 2 of 5 probes diverged between arms. On re-test one of the two came back byte-identical. The surviving difference was two defensible implementations (one returned {} on empty input, the other raised TypeError on a non-string; one added comments). A reviewer asked to rank them would have no basis.

Reading vLLM's sampler settles what this is not. Rejection sampling runs in "standard" mode, the provably lossless algorithm, and at temperature 0 the acceptance test is literally an argmax comparison: mathematically exact. What the proof assumes and hardware does not provide is exact arithmetic. MTP's verification pass processes two positions instead of one, which selects a different CUDA kernel, which changes floating-point reduction order, which flips last bits in the logits. When two candidate tokens are near-tied, the argmax flips, and greedy decoding forks permanently. INT4 amplifies this: quantization compresses the logit distribution and manufactures near-ties.

Temperature is the detector here, not the cause. Production runs at temperature 0.7, where consecutive identical requests already differ; a perturbation below that sampling noise has no "correct" output to deviate from.

The correction, and the question still open

The MTP quality post claimed that divergence at temperature 0 "is a verification bug by definition." That was too strong, and it now carries a correction note. Temperature-0 divergence proves outputs change. It does not, by itself, prove they degrade. There are two mechanisms on the table, and they predict the same hashes:

Benign: the near-tie coin-flip above, with no directional bias; a near-tie resolved the other way is not a worse token. Not benign: this model is 48 of 64 layers Gated DeltaNet, the architecture whose recurrent state cannot currently be rolled back correctly on draft rejection (vLLM PR #51508, open, and absent from the deployed version; the relevant state-update code shows no zero-accept guard). The evidence, both columns stated fairly:

Our measurements cannot distinguish these, and the honest state is not "unmeasured risk" but "a specific, named corruption mechanism was not ruled out." The discriminating experiment is also specific: repeat the temperature-0 protocol across draft depths and test whether divergence is depth-dependent (best run on the sibling model where multiple depths exist), and score both arms on an execution-graded benchmark, which the Ornith work showed is the only arbiter that matters. Until then the deployment's status is stated in its own docs in exactly these words: enabled on a throughput result, not a quality result. The developers using it are the quality monitor, and the rollback is one file copy.

Takeaways

• Blind trust and blind fear of MTP are the same mistake. The throughput crowd enables it without scoring quality; anyone who has seen a divergence report disables it without asking which mechanism produced the divergence. Both skip the measurement.
• Quality-first draft depth is modest. This head is trained for exactly one step, our halo tuning landed on n=1 against a community default of 3, and acceptance has collapsed with depth in every measurement this series has made.
• Judge speculative decoding by end-to-end wall time, not decode rate or TTFT. Here TTFT doubled, decode rose 13-35% depending on concurrency, and the number that matters, the full round trip at production completion lengths, improved 30-40% with 3x less variance.
• Know your workload's shape. This endpoint's engine time is 77% decode (prefix caching absorbs most prefill), and completions are short. That is the shape where a decode-side accelerator pays end to end. The same flag on a prefill-dominated workload buys much less.
• Proofs of losslessness assume exact arithmetic. Real kernels reduce in different orders under different batch shapes. "Provably identical" and "bitwise identical" are different claims, and INT4 widens the gap.

Environment: official vLLM v0.27.1 image (verified unmodified by digest; spec-decode files byte-identical to the public tag), one H100 80GB SXM, RedHatAI/Qwen3.8-27B-INT4, method mtp, k=1. Measurement: client-side wall time over n=10 per arm at concurrency 4; acceptance and KV figures from vLLM's own metrics. Series context: MTP quality cost, Ornith tuning, H100 production.

The day-one post established that Qwen3.8-Flash-Next runs on this box. This one is the campaign that followed: tune it, build the missing backend, measure quality across quants, and find out where the speed went. The short version: production candidate is IQ3_XXS at -ub 2048 --parallel 2 on a fresh master build, and the biggest single performance finding cost zero lines of code, because it had already been merged upstream by people who never heard of this machine.

Tuning: two levers paid, two didn't

Method as in the tuning post: one variable at a time, 3-repeat noise floor, journaled runs, via strix-halo-optimize.

• -ub 2048: +13-28% prefill on every quant at every depth, decode flat, plateaus past 2048. (On this model the sweet spot is 2048, not the 1024 that won on Qwen3.6; calibrate per model.)
• --parallel 2: +65% aggregate decode across two streams. The multi-slot desync bug from the day-one post is confirmed fixed.
• PLE/tensor placement flags: no-op. On 128 GB unified memory everything already fits at -ngl 999; these levers exist for VRAM-constrained discrete GPUs.
• mmap: a 20-30% regression. --no-mmap stays mandatory on this box, this model included.

One measurement reframed the whole campaign. At ~22 t/s decode, this model uses roughly a quarter of the memory bandwidth its active parameters predict (~84 t/s theoretical), while rocm-smi shows the GPU 78-96% busy throughout. Decode here is compute-bound in the kernels, not memory-starved. That is unusual for MoE decode on this hardware, and it pointed at the kernel-profiling phase below.

The first working Vulkan build, and a quant-dependent crossover

No Vulkan build of the qwen4exp architecture existed publicly, so one got assembled: the exact source branch reverse-engineered from the git history of kyuz0's retired toolbox, combined with the standard Vulkan build recipe. Across four quants (Q3_K_XL, IQ3_XXS, Q2_K_XL, IQ1_M):

• Vulkan wins decode on every quant, +10-14%, no exceptions.
• Prefill depends on quant size: ROCm wins on the two larger quants (up to +14.5%), Vulkan on the two smaller ones.

The running series lesson (backends are per model family) gains a refinement: sometimes they are per quant, too.

Quality across quants: a cliff, not a slope

Quality was scored with to-bench, the execution-based harness that grew out of the MTP quality work: real coding prompts graded by running hand-written pytest suites against the model's code, no LLM judge anywhere, partial credit, identical seeds across configs. It is now a 52-task tool with a locked protocol, due for public release shortly.

Two false starts first, both caught by the protocol rather than luck: an initial run on the easy task set (saturates near 100% on any competent model, discriminates nothing, the same ceiling effect that fooled the MTP eval's first phase), and an initial run at the wrong reasoning setting for cross-cohort comparability. Re-run on the full 52 tasks, 5 seeds, locked protocol:

The non-monotonic row is not noise: IQ1_M beats the larger Q2_K_XL because two specific tasks fail 5 seeds out of 5 at Q2_K_XL, a reproducible quality cliff at that precision on those tasks. Same shape as the Ornith quant findings: quantization damage is per-task and uneven, not a smooth slope down the bit-width ladder.

The keeper is IQ3_XXS: statistically tied with Q3_K_XL (the intervals overlap heavily) at 8.7% smaller, with faster two-stream throughput than any larger option.

The profiling payoff: the fixes already existed

Why is decode compute-bound? An execution-graph dump (llama-eval-callback, ships in the toolbox) confirmed the op mix by name: GATED_DELTA_NET, SSM_CONV, MUL_MAT_ID. That matched llama.cpp issue #20354 almost exactly: GDN kernels running ~4x slower than expected on ROCm, never re-tuned for RDNA 3.5, reported on this same hardware family.

Then the actual finding, one layer deeper: this was already fixed. A HIP shared-memory fix for the GDN kernels (#20366) and a Vulkan GDN shader (#20334) both merged in mid-March, and qwen4exp itself merged to master on 2026-08-27, after both. Our custom build was hand-rebased from a pre-merge branch and never had a chance to combine all three. So instead of writing a patch: rebuild both backends from current master and re-measure. Vulkan gained +15-16% decode, ROCm +9% short-context prefill and +5% short-context decode. Free speed, zero patch-writing, correctness re-verified on the new build.

The lesson generalizes to anyone running day-one models: a pre-merge build ages backwards. Every week it lives, it silently misses mainline fixes, and the first debugging step for "this architecture is slow" is not a profiler, it is a rebuild.

Giving back, and two incidents

Two open llama.cpp PRs explicitly wanted testers on this GPU family, so both got built and benchmarked against the master baseline, with numbers posted upstream: one (GDN cache-write fusion) neutral within noise here, the other (a chunked-prefill kernel) a 16-24% prefill regression on this hardware, opposite the author's +30% on a different device. Negative results on the right hardware are exactly what draft PRs need.

And the incident file grew. The stale-server bug bit again in a new costume: kill $SERVER_PID on a toolbox-wrapped llama-server kills the wrapper, not the podman-exec'd server inside, which silently corrupted one eval run against a leftover model before a /v1/models check caught it, mid-run. That is the third measured-the-wrong-thing incident this series (the judge, the benchmark arms, now this), and the countermeasure is now reflexive: verify what is actually serving before trusting any result. Separately, a research subagent that was asked to investigate profiler tooling instead attached to and relaunched the live production server, and the only reason it was caught is that production state got verified directly rather than taking the agent's self-report at face value. Agents inherit the stale-server lesson too: trust state you checked, not state you were told.

Bottom line

Production candidate: IQ3_XXS, -ub 2048 --parallel 2, rebuilt from current master, Vulkan for decode-heavy use and ROCm for prefill-heavy at this quant tier. Not done yet, stated plainly: the quant sweep and the quality eval have not been re-run on the new master builds, so the numbers above are internally consistent on the old builds but pre-date the free upstream speed. That re-run is the last gate before calling this final.

Setup and history: day-one post, base stack, tuning methodology repo.

Qwen3.6-35B-A3B is this box's production workhorse: Q8_0, served at --parallel 6 with 256K context split across agents, because the primary workload here is four-plus parallel coding agents, not single-user chat. A tuning session on it produced one clean performance win and one uncomfortable reliability finding. The second corrects advice this blog published.

The method behind both was the same: one variable at a time, real before/after measurement on this exact model at this exact concurrency, and no community number (or previously written-down claim) trusted over an actual measurement. The benchmarks wrapped llama.cpp's own tools, with provenance (kernel, container digest, exact command) captured per run.

Finding 1: -ub 1024

Correction, same day: the first version of this post claimed +25-30%, comparing runs taken on opposite sides of the GPU power-state change described in Finding 2. That state change is itself worth 2.4-4.9% on this machine, so the comparison was confounded and the number inflated. The table below is the clean same-state comparison (both runs at the same governor setting, three minutes apart). The append-only findings log that caught this is the subject of the next post.

Production had never set the ubatch size, so it ran llama.cpp's default of 512. Raising it to 1024, measured across the real concurrency curve with q8_0 KV cache, aggregate prefill:

The gain is remarkably uniform, +21.5% to +22.3% across every row with a same-state baseline (the baseline run died 6 rows in when the machine crashed, so pp8192 at c=4/c=6 lack a clean before). Decode stayed flat within the noise band, exactly as expected: ubatch governs prefill compute-buffer sizing and does not touch the decode path. For this workload, prefill is the number one axis; coding agents re-send long, growing contexts every turn. One flag, ~22%, no downside. Applied to the production conf, validated with a live completion and a tool-call test through the real llama-swap serving stack, shipped.

Finding 2: the forced-high GPU governor, retracted

Mid-session, the machine hard power-cut, the second such crash in days and the seventh since April, with the cadence accelerating: two in two months, then four or five in three weeks. The signature is identical every time. The kernel log goes silent, the journal keeps ticking routine heartbeats, then simply stops: no panic, no shutdown sequence, which is the classic shape of a firmware-level critical cut before the OS can log anything. The two most recent crashes both carried an "internal CPU thermal limit was tripped" flag in the AMD platform reset-reason register, with the caveat that this register is sticky and cumulative, so it is supporting evidence, not proof.

The root-cause candidate surfaced during the investigation: gpu-power-high.service, which locks power_dpm_force_performance_level to high at boot. Checking the actual clock tables showed what that really does on this machine: GPU clock pinned at a fixed 2900 MHz around the clock, at idle, not just under compute.

That service exists because this blog put it there. The gotchas post and the setup guide both recommend forcing high persistently, on the claim that the auto governor costs up to 50% throughput. Here is the uncomfortable part: that number came from community documentation, and going back through my own notes, I never measured it. No before/after exists. The tuning skill's own reference material flags forcing high as a diagnostic, not a tune, and my config had it locked anyway, on an unverified claim.

Switching to auto and disabling the service had an immediate, measured effect: idle package power dropped from 27-29W to 11W, roughly 60% less standing power and heat, on a machine that idles most of every day. Whether auto costs any throughput under real load is now an open measurement (DPM should boost clocks under sustained compute on its own), and it is next on the list, but the direction of the error is already clear: I was paying a continuous thermal and power tax for a benefit I had never verified, on a machine with an accelerating hard-crash habit whose evidence points thermal. Both published pages now carry a retraction note.

Benchmarking resumed only after the machine was inspected, with live kernel-fault monitoring and continuous sensor logging running throughout; temperatures stayed nowhere near limits for the rest of the session.

The takeaway

Same discipline, two payoffs in one session: a 22% prefill win from measuring a default nobody had questioned, and a reliability fix from finally measuring a "fact" this blog itself had published. Measurements beat documentation, and that includes your own documentation. The unverified claim you wrote down a few months ago is exactly as suspect as the unverified claim you found on a forum, and considerably more dangerous, because you trust it more.

Reproducibility: AMD Ryzen AI MAX+ 395 (Radeon 8060S, gfx1151), 128 GB unified memory, ROCm 7.2.4 container, Qwen3.6-35B-A3B Q8_0, -fa 1 --no-mmap -ngl 999 -ctk q8_0 -ctv q8_0, llama-batched-bench sweeps at pp 2048/8192, tg 128, concurrency 1/2/4/6. The change shipped to production: -ub 1024. Governor: auto (default), gpu-power-high.service disabled.