MTP - multi-token prediction, the speculative decoding where a model drafts its own future tokens with a built-in head instead of a separate draft model - has had a rough arc on this blog: a tuning win that required rejecting the community default, a measured quality cost on hard tasks, a cross-engine divergence confirmation at temperature 0. The running advice was: measure before you trust it.

Soo... we measured. A lot. The result is that halo now runs two MTP routes in production: Gemma-4-26B at +69% decode, and Qwen3.8-Flash-Next at a campaign-validated +51% mean. This post is how they earned their way in, including the backend that got banned and the claim of ours that did not survive its own validation campaign.

Gemma-4: the easy one

Unsloth publishes an official MTP draft sidecar for Gemma-4-26B-A4B: a 441 MB "smart Q4_0" drafter that llama.cpp has supported since June. Wire it in with --spec-draft-model and --spec-type draft-mtp, and on this box: 40.88 to 68.97 t/s on a short real prompt (+68.7%, 78.7% acceptance), 70.68 t/s sustained through a 2500-token generation at 82% acceptance, no collapse, no corruption.

Correctness got the treatment the quality-cost findings made mandatory: temp-0 comparison (not byte-identical, as expected by now - more below), then the generated code extracted and executed against real test cases. Passed.

One deployment decision worth copying: the MTP route is a separate, opt-in, single-slot route (gemma-4-26b-a4b-mtp), not a replacement for the production multi-slot route. Why? MTP gains invert under concurrency on this hardware - measured for the Qwen3.6 family, inherited here as the safe default rather than re-proven for Gemma-4 specifically. Batching already amortises what speculation buys; the H100 team found the same shape independently. Single-stream lever, single-stream route.

Flash-Next: the hard one

Qwen3.8-Flash-Next ships a native MTP head, but llama.cpp support for it still lives in open PRs (#27836 base support, unmerged as of this writing). The working path came from the community: a member (drluoto) assembled a branch combining the PR head with the loader and tensor-naming fixes needed to actually run it, and that branch became a custom ROCm toolbox here. Our base quant (UD-IQ3_XXS) paired with the Q8_0 sidecar was a combination nobody had published numbers for.

The first smoke test looked almost too good: 24.84 to 45.30 t/s, +82%, 97.2% acceptance. House rules say a single prompt adopts nothing, so it got a campaign: 5 real to-bench task prompts × 3 seeds, MTP-on vs MTP-off, paired. Result: +51.1% mean speedup (range +36.2% to +60.7%, stdev 8.2%), acceptance 87.4%, consistent across every pair. Then a third, independent cross-check on fresh real prompts that were never part of the campaign: +38.5% and +57.0%, landing inside the campaign's range. Three methods, one effect size. That's an adoption.

And a swap worth a paragraph: when Unsloth later published an official sidecar for this model, it got A/B'd against drluoto's on identical prompts. 40.55 vs 39.59 t/s, 90.5% vs 89.5% acceptance - essentially equivalent, slight edge to official. Adopted anyway, for the boring reason: a maintained official source beats a community rehost that could go stale. The community sidecar stays on disk as rollback. Community work got us here months early; official sources are what you settle on.

The backend that got banned

This box's default backend preference is Vulkan RADV for most things. For MTP it inverted, and the way it inverted is the cautionary tale.

On a short 143-token prompt, Vulkan MTP looked like a clean win: +60%, 94.4% acceptance. On a longer, harder 2000-token generation, throughput collapsed to a steady 6.3-6.6 t/s - not a crash, a collapse, sitting at a fifth of the no-speculation baseline (29.33 t/s) from early in the generation to the end. A short-prompt-only test would have shipped a severe regression into production with a green checkmark on it.

Honest scoping: the short and long tests differed in context size and prompt content, not just length, so the exact trigger wasn't isolated. But the operational conclusion needed no more precision: ROCm is the only backend validated safe for MTP on this hardware, through full-length generations. Benchmark depth is not optional. It's where this class of failure lives.

The claim we had to correct

That first Flash-Next smoke test also reported MTP-on and MTP-off producing byte-identical output at temperature 0. The validation campaign proved our own claim wrong: 0 of 15 pairs were byte-identical. Small floating-point-driven divergence appeared in every single pair; the smoke-test prompt just happened not to trigger it.

What was preserved is the thing that matters: functional correctness. Five pairs hit token-budget truncation in both arms and weren't comparable; of the comparable pairs, all but one had identical pytest pass/fail outcomes on the extracted code, and the one divergent pair scored better with MTP on (12/13 vs 10/13) - verified by full pytest diff to be a genuinely different-but-valid implementation, not corruption.

This slots cleanly into the two-mechanisms picture this blog ended up with: divergence is universal (the losslessness proof assumes exact arithmetic, hardware disagrees), and whether it costs anything is a per-model, per-head question only an executed benchmark can answer. Ornith's immature head cost 17-25% on hard tasks. Flash-Next's, on this evidence, costs nothing measurable. Both posts now cite each other, because readers of either need the other half.

Where this lands

I'd put the state of MTP on this hardware like this: it is neither the free speedup the leaderboards imply nor the quality hazard a divergence report implies. It is a per-model, per-backend, per-workload lever that pays +40-70% single-stream when the head is well-trained, inverts under concurrency, collapses on the wrong backend, and always changes outputs at the bit level without necessarily changing them at the level that matters. Every clause in that sentence was measured on this box, most of them twice.

The validation tax for the two adoptions was real: campaigns, cross-checks, temp-0 diffs, executed code. Also, I'd say, obviously worth it: the alternative was either leaving 50-70% of decode on the table or shipping a Vulkan collapse. Measure before you trust it still stands. It's just that now, twice, the measurement said yes...