Every speed number this blog has published for halo rests on llama-bench, and llama-bench's prompts are synthetic: repeated tokens, not real text. A reader raised the reasonable question - do those figures reflect real usage at all, or are they an artifact of the corpus? Fair challenge. The answer to a fair challenge is a measurement, not a defence.

The cross-check: real prompt content (concatenated coding-task prompts, reference solutions, and a genuine coding question) at three sizes - 206, 3,192, and 6,421 real tokens - run three times each on Qwen3.8-Flash-Next UD-IQ3_XXS, on both ROCm and Vulkan backends, and compared against the synthetic llama-bench sweeps at the corresponding depths. Decode measured the same way: 250 real generated tokens on the same prompts.

The result, backend by backend

ROCm: essentially an exact match. The real 3,192-token prompt prefilled at 392.98 t/s. The synthetic pp4096 sweep: 390.78 t/s. That's 0.6% apart, within run-to-run noise. The 6,421-token real prompt (371.10 t/s) fits smoothly on the synthetic declining-with-depth curve. Decode: real-prompt numbers land 4.9% from the synthetic figure at comparable depth, with rep-to-rep spread of 0.05-0.21 t/s. Hard to ask for better.

Vulkan: a real 8.7% gap, disclosed rather than papered over. Real 3,192-token prefill measured 344.09 t/s against synthetic pp4096's 377.07. The plausible explanation: llama-bench's chunked-at-depth measurement and a fresh single-shot prompt of comparable length are not strictly identical measurements - attention cost accumulates differently - so the gap doesn't necessarily mean either number is wrong. But it's real, it's backend-specific, and if you quote Vulkan synthetic numbers, real prompts on this box run a bit slower than they suggest.

One number that looks alarming and isn't: the short 206-token real prompt measured far below the synthetic short-depth figure (182.63 vs 431-ish on ROCm). Not a discrepancy - llama-bench's "pp0" measures a 2,048-token chunk, and small-batch fixed overhead dominates a 206-token prompt. Not a like-for-like comparison, so don't read it as one.

Worth saying why this check mattered extra on this particular model: it's a MoE, and expert routing is content-dependent. Synthetic repeated tokens could, in principle, exercise a completely different expert-load pattern than real code. Measured verdict: they don't, at least not enough to matter on ROCm.

The bug the check itself caught

The first attempt at this cross-check produced garbage, and the way it did is the transferable lesson. Reps 2 and 3 of each prompt came back suspiciously fast - because llama-server's prompt cache silently recognised the repeated prompt and collapsed the prefill to a handful of tokens. The "benchmark" was measuring cache lookups. Caught by inspecting the per-rep prompt_n fields, fixed with cache_prompt:false, whole exercise redone.

That's the third measured-the-wrong-thing incident this series (the judge, the stale server, now the cache), and I'd say the pattern is now a rule: any live-serving benchmark must either disable prompt caching or vary the prompt per rep, and you verify which happened from the server's own counters, not from the wall clock looking plausible.

Where this leaves the numbers

So: synthetic benchmarks, on this box, on this model class, measure the real thing - on ROCm nearly exactly, on Vulkan with a known, quantified offset. The MTP campaign numbers got the same treatment (fresh real prompts, never used in the original campaign) and converged on the same effect size, which is what let those adoptions ship with confidence.

The general habit I'd recommend from all this: don't defend your benchmarks, cross-check them. It cost one afternoon, it answered the skeptic properly, and it found a methodology bug that would have quietly poisoned some future measurement anyway...