Qwen3.8-27B-INT4 with MTP: 30-40% faster round trips, and two rival explanations for the divergence

This blog's MTP scoreboard so far: a throughput win on Strix Halo that required rejecting the community default, a measured quality cost on hybrid-GDN models in two engines, and a "don't enable it" verdict on the H100's MoE workload. This post adds the first entry where MTP went into production: the H100 endpoint moved to an INT4 checkpoint of Qwen3.8-27B, and that swap unlocked a usable MTP head. The numbers say run it. The mechanism story says keep watching, and it corrects something this blog previously stated too strongly.

Why the quant swap unlocked MTP: the INT4 source ships its 849 MB MTP head in BF16, explicitly held out of quantization. One trained layer, so draft depth 1 is the only native depth. Regular readers will recognize that as the depth our halo tuning independently converged on.

The number that matters: end-to-end request time

Not decode rate. Full HTTP round-trip from the client, TTFT included, at concurrency 4 with ~36K-token prompts and 145-token completions (145 is production's mean completion length; the median is 20-50 tokens, and 89% of real completions are under 200):

n=10 per arm, mean ratio 1.41x, median 1.30x, t=3.91. The variance drop is arguably the better half of the result: a 3x tighter spread is what an interactive agent user actually feels.

The counterintuitive part worth reporting: TTFT roughly doubled (0.61 s to 1.2 s), and total time still fell. The verification pass makes the first token more expensive; every step after it yields more than one token; over even a short completion the repayment wins. If you monitor TTFT as your latency proxy, MTP will look like a regression while making every request faster.

Caveats, stated plainly: synthetic shared-prefix prompts at 36K (production averages are longer), an otherwise idle box, and one earlier n=1 baseline measurement (2.90 s) that contradicts the n=10 distribution and is disclosed rather than hidden.

Why acceptance is high here, and what it cost

Draft acceptance ran 93-94% at 400-token completions and 82-88% on the short reps. The series now has an acceptance ladder, and it tracks how purpose-built the drafter is:

Costs: the KV pool shrank 9.1% (max concurrency at full context 6.36x to 5.78x), mild next to the 41.6% the DFlash drafter cost. Two sharp edges worth copying into your own checklist: min_p and logit_bias silently stop working under speculative decoding in vLLM (it warns once at boot, then fails quietly), and one known open vLLM spec-decode bug is present in this image version but did not fire in testing.

The divergence, measured

At temperature 0, five coding probes, three repeats per arm: both arms fully self-stable, and 2 of 5 probes diverged between arms. On re-test one of the two came back byte-identical. The surviving difference was two defensible implementations (one returned {} on empty input, the other raised TypeError on a non-string; one added comments). A reviewer asked to rank them would have no basis.

Reading vLLM's sampler settles what this is not. Rejection sampling runs in "standard" mode, the provably lossless algorithm, and at temperature 0 the acceptance test is literally an argmax comparison: mathematically exact. What the proof assumes and hardware does not provide is exact arithmetic. MTP's verification pass processes two positions instead of one, which selects a different CUDA kernel, which changes floating-point reduction order, which flips last bits in the logits. When two candidate tokens are near-tied, the argmax flips, and greedy decoding forks permanently. INT4 amplifies this: quantization compresses the logit distribution and manufactures near-ties.

Temperature is the detector here, not the cause. Production runs at temperature 0.7, where consecutive identical requests already differ; a perturbation below that sampling noise has no "correct" output to deviate from.

The correction, and the question still open

The MTP quality post claimed that divergence at temperature 0 "is a verification bug by definition." That was too strong, and it now carries a correction note. Temperature-0 divergence proves outputs change. It does not, by itself, prove they degrade. There are two mechanisms on the table, and they predict the same hashes:

Benign: the near-tie coin-flip above, with no directional bias; a near-tie resolved the other way is not a worse token. Not benign: this model is 48 of 64 layers Gated DeltaNet, the architecture whose recurrent state cannot currently be rolled back correctly on draft rejection (vLLM PR #51508, open, and absent from the deployed version; the relevant state-update code shows no zero-accept guard). The evidence, both columns stated fairly:

Our measurements cannot distinguish these, and the honest state is not "unmeasured risk" but "a specific, named corruption mechanism was not ruled out." The discriminating experiment is also specific: repeat the temperature-0 protocol across draft depths and test whether divergence is depth-dependent (best run on the sibling model where multiple depths exist), and score both arms on an execution-graded benchmark, which the Ornith work showed is the only arbiter that matters. Until then the deployment's status is stated in its own docs in exactly these words: enabled on a throughput result, not a quality result. The developers using it are the quality monitor, and the rollback is one file copy.

Takeaways

• Blind trust and blind fear of MTP are the same mistake. The throughput crowd enables it without scoring quality; anyone who has seen a divergence report disables it without asking which mechanism produced the divergence. Both skip the measurement.
• Quality-first draft depth is modest. This head is trained for exactly one step, our halo tuning landed on n=1 against a community default of 3, and acceptance has collapsed with depth in every measurement this series has made.
• Judge speculative decoding by end-to-end wall time, not decode rate or TTFT. Here TTFT doubled, decode rose 13-35% depending on concurrency, and the number that matters, the full round trip at production completion lengths, improved 30-40% with 3x less variance.
• Know your workload's shape. This endpoint's engine time is 77% decode (prefix caching absorbs most prefill), and completions are short. That is the shape where a decode-side accelerator pays end to end. The same flag on a prefill-dominated workload buys much less.
• Proofs of losslessness assume exact arithmetic. Real kernels reduce in different orders under different batch shapes. "Provably identical" and "bitwise identical" are different claims, and INT4 widens the gap.

Environment: official vLLM v0.27.1 image (verified unmodified by digest; spec-decode files byte-identical to the public tag), one H100 80GB SXM, RedHatAI/Qwen3.8-27B-INT4, method mtp, k=1. Measurement: client-side wall time over n=10 per arm at concurrency 4; acceptance and KV figures from vLLM's own metrics. Series context: MTP quality cost, Ornith tuning, H100 production.