MTP on Strix Halo: +79% at one stream, zero at four, and the ceiling isn't bandwidth after all

This blog's running finding on MTP - speculative decoding via a model's built-in multi-token-prediction head - has been that its gains invert under concurrency: measured on the halo box at high slot counts, measured on the H100 where MTP was outright slower than baseline at 8 concurrent streams. The safe deployment shape we settled on was single-slot, opt-in routes.

New measurement, cleaner than any of the earlier ones, and it refines the doctrine. Qwen3.8-Flash-Next (UD-IQ3_XXS, its MTP sidecar, the newly adopted engine build), real coding generations rather than synthetic tokens, thinking off so tokens are code tokens, speeds read from the server's own timings. Aggregate throughput:

Draft acceptance at 2 slots: 98%. Correctness under concurrency: 18 of 18 answers correct at both 2 and 4 slots (6 distinct prompts, 3 rounds each). And the row worth staring at is the last one: with or without MTP, four concurrent streams aggregate to the same ~63 t/s.

What the ceiling is, and what it isn't

Correction, same day: the first version of this section called the plateau "the signature of a memory-bandwidth-bound machine". A reader's comment on the announcement took that apart, and he's right. What follows is the corrected analysis; the original claim is retracted.

The counterargument is self-contained and worth learning from. If the binding cost were bytes of weights per second, then amortising weight reads - which is exactly what both MTP and batching do - should keep raising aggregate tokens per second. A ceiling that is indifferent to how much you amortise is not a per-weight-pass cost. It is a per-token cost. My own explanation ("speculation and batching are the same trick") was, read carefully, an argument against my own headline.

And our own numbers agree, once actually computed instead of gestured at. Plain decode moves roughly the model's ~6B active parameters per token, call it 2-3 GB at this quant; at 27.8 t/s that is ~70 GB/s of achieved bandwidth, about a third of the ~215 GB/s this box can move. Not bandwidth-bound, not even close. The step times say the same: a 1-token step costs ~36 ms, a 4-token verification step ~63 ms. If weights were read once and shared, the ratio would be near 1x, not 1.75x. And an earlier quant sweep implies a fixed ~18 ms floor per decode step regardless of model bytes, roughly 40% of the step.

So what is the per-token cost? Honestly: not established. It does not look like a clean FLOPS roofline either (the arithmetic puts utilisation well below peak). The live suspects are per-token kernel work that amortises poorly: 36 of this model's 48 layers are recurrent gated-delta-net, whose state updates cost per token verified; MoE routing means each drafted token can hit different experts, so expert bytes grow with draft length (the amortisation premise was never quite true for a MoE); and IQ3_XXS is one of the most dequant-heavy formats there is. Hypotheses, all three. The settling experiments are named and queued: profile achieved GB/s during a 4-slot run, and repeat it on a cheap-to-dequant quant like Q4_K - if the ceiling rises despite more bytes, it was never about bytes.

What survives untouched is the contrast with the H100 result, where the same lever went negative under load because that box binds on KV-cache headroom and scheduler pressure, which MTP's overhead actively damages. Halo's ceiling absorbs the overhead; the H100's constraints are hurt by it. So the transferable rule stands, now with better epistemics: MTP's value under concurrency isn't a property of MTP. It's a property of your machine's binding constraint - and you should identify that constraint by measurement, which, as this correction demonstrates, I had not.

What changed in production

The old doctrine (single-slot MTP routes only) was inherited from measurements where gains inverted. This curve says something friendlier for this box: the gain tapers, but it never goes negative, and at 2 slots it's still +67%. So the production route for this model is now --parallel 2 with MTP - two agents get 63.5 t/s aggregate where they used to get 38. MTP stays on because on this box there is, on this evidence, no concurrency level at which it hurts - a measured fact that does not depend on knowing why the ceiling sits where it does. Quality: unchanged, as previously established - same weights, verified drafts, matching perplexity, and the earlier campaign's 0-of-15 byte-identical / 10-of-11 functionally-identical result still stands.

Honest limits

Mostly single runs (the winner's repeat agreed within ~4%), one machine, one quant, up to 4 slots. Not tested: the full 262K context, more than 4 slots, quantized KV combined with MTP, or a long soak under real agent load. The engine underneath is an experimental community build, hand-built and unpinned - the fuller audit that selected it is its own story. And "never negative" is a claim about this box's constraint, not about yours: on a machine that binds on KV or compute first, the H100 numbers are the ones to expect.

The MTP series scoreboard, five posts in: it changes outputs at the bit level everywhere, costs quality on some heads and none on others, pays +40-80% single-stream when the head is good, and under concurrency it does whatever your bottleneck tells it to. Measure the bottleneck first - actually measure it, not name it from vibes as the first version of this post did - and the MTP decision makes itself...