MTP on Strix Halo meets the bandwidth ceiling: +79% at one stream, zero at four, never negative

This blog's running finding on MTP - speculative decoding via a model's built-in multi-token-prediction head - has been that its gains invert under concurrency: measured on the halo box at high slot counts, measured on the H100 where MTP was outright slower than baseline at 8 concurrent streams. The safe deployment shape we settled on was single-slot, opt-in routes.

New measurement, cleaner than any of the earlier ones, and it refines the doctrine. Qwen3.8-Flash-Next (UD-IQ3_XXS, its MTP sidecar, the newly adopted engine build), real coding generations rather than synthetic tokens, thinking off so tokens are code tokens, speeds read from the server's own timings. Aggregate throughput:

Draft acceptance at 2 slots: 98%. Correctness under concurrency: 18 of 18 answers correct at both 2 and 4 slots (6 distinct prompts, 3 rounds each). And the row worth staring at is the last one: with or without MTP, four concurrent streams aggregate to the same ~63 t/s.

The ceiling does the explaining

That convergence is the signature of a memory-bandwidth-bound machine. Halo moves roughly 220 GB/s, and for this model that budget buys about 63 tokens per second of aggregate decode, however you spend it. Speculative decoding and multi-slot batching are, at bottom, the same trick: amortise each pass over the model's weights across more than one emitted token. One stream with MTP, four streams without, or two streams with both - all roads converge on the ceiling. Once you're there, speculation has nothing left to buy, because the resource it saves is already fully spent.

Which also explains, by contrast, why the H100 result went negative under load while halo merely flattens. On the H100 the binding constraints were different - KV-cache headroom (the drafter cost 9% of it) and scheduler pressure - so MTP's overhead had something to actively damage. On halo the overhead simply disappears into a bandwidth budget that was going to be exhausted anyway. Same lever, opposite failure modes, decided entirely by which wall the hardware hits first. I'd say that's the transferable rule: MTP's value under concurrency isn't a property of MTP. It's a property of your machine's binding constraint.

What changed in production

The old doctrine (single-slot MTP routes only) was inherited from measurements where gains inverted. This curve says something friendlier for this box: the gain tapers, but it never goes negative, and at 2 slots it's still +67%. So the production route for this model is now --parallel 2 with MTP - two agents get 63.5 t/s aggregate where they used to get 38. MTP stays on because on a bandwidth-bound machine there is, on this evidence, no concurrency level at which it hurts. Quality: unchanged, as previously established - same weights, verified drafts, matching perplexity, and the earlier campaign's 0-of-15 byte-identical / 10-of-11 functionally-identical result still stands.

Honest limits

Mostly single runs (the winner's repeat agreed within ~4%), one machine, one quant, up to 4 slots. Not tested: the full 262K context, more than 4 slots, quantized KV combined with MTP, or a long soak under real agent load. The engine underneath is an experimental community build, hand-built and unpinned - the fuller audit that selected it is its own story. And "never negative" is a claim about this box's constraint, not about yours: on a machine that binds on KV or compute first, the H100 numbers are the ones to expect.

The MTP series scoreboard, five posts in: it changes outputs at the bit level everywhere, costs quality on some heads and none on others, pays +40-80% single-stream when the head is good, and under concurrency it does whatever your bottleneck tells it to. Measure the bottleneck first and the MTP decision makes itself...