A Strix Halo engine audit: a scary claim that didn't reproduce, a fork worth +35-92% prefill, and seven broken routes

A research sweep of the Strix Halo ecosystem turned up four things at once: newer kyuz0 toolbox builds, an experimental toolbox running a Halo-specific llama.cpp fork on a patched ROCm runtime, a community author who abandoned ROCm claiming it "returns wrong logits for any prompt longer than the batch size" (perplexity 84.8 versus 13.8 on Vulkan), and a headline of 1207 t/s prefill on this hardware. House rule, by now well earned: nothing gets adopted on faith. So everything got tested. This is the short version; the full scripts and results are journaled in the audit's own directory.

The scary claim: not reproduced

Wrong logits from a production backend would outrank every speed number on this blog, so it went first. Method: llama-perplexity over 160 KB of real Python source, same text and model on both backends, varying exactly the thing the claim names (physical batch size smaller than vs equal to the prompt) plus the claimed workaround (HIP_LAUNCH_BLOCKING=1). Two models: the production Qwen3.6-35B and Qwen3.8-Flash-Next.

Every ROCm configuration landed within 0.3% of Vulkan. Qwen3.6: 1.2994 on ROCm in all three configs against Vulkan's 1.2981. Flash-Next: 1.1750-1.1754 against 1.1718. We also re-ran the concurrency-corruption check from the known HIP bug: 12 of 12 concurrent answers correct. Stated carefully, because this matters: we cannot prove the claim wrong on the author's build and workload. We can say it does not reproduce on ours, with the exact knob it names being turned. Perplexity that moves 0.3%, not 6x, is not silent corruption.

The engine race: real prompts, one winner

Flash-Next's engine has been a hand-rebased fork since day one. Three candidates ran the same weights, same MTP sidecar, same inputs: prefill on real source code at three depths with caching off, decode on real coding tasks with thinking off, numbers from the server's own timings.

Adopted: strix-llama. Roughly +35-92% prefill and +24-30% decode over the old engine, on the metric this box actually cares about. The asterisk is a measurement gotcha worth stealing: the first request after model load reads low. Steady-state 2k prefill on the adopted engine is ~636 t/s, not the 300-520 the first request shows, so our own first comparison understated the winner. If your benchmark's first row looks odd, it probably is.

The 1207 t/s headline? A different measurement: synthetic pp2048 at depth zero, a different quant, -ub 16384. Our best real-prompt number is 924 t/s at 24k depth. Not reproduced on our metric is not the same as false, and this one is a textbook case of why benchmark configs must travel with benchmark numbers.

The Vulkan runner-up lost on everything except decode: slower prefill at depth (the wrong direction for agentic work), 439 commits behind upstream (which means older tool-call and template handling, exactly what agents exercise), and a trimmed sidecar format incompatible with the official one, so its speedups can't be combined with anything else's. Credit where due: it fixed the September Vulkan MTP collapse, holding 55 t/s over a 1037-token generation where the old branch fell to 6.3. Quality on the adopted engine: unchanged by every check we have (matching perplexity to the fourth digit, identical outputs on repeated prompts, 5/5 generated-code tests, 4/4 tool-calling round trips in both thinking modes).

The war story: a routine refresh broke seven routes

The same sweep refreshed two standard toolboxes, and the new builds removed --no-mmap outright in favour of --load-mode. Our launcher passed --no-mmap by default. Seven model routes, including the always-on model behind the assistant's research sidecar and the default agentic coder, failed to start for about twenty minutes, and the only reason it was twenty and not a day is that one real test request went through after the refresh. The container's version string looked fine throughout. This is the same flag removal that broke the tuning skill's scripts two weeks ago, now graduated from "migration note" to "production incident."

The fix is a probe, not a pin: the launcher now asks the container's llama-server whether it speaks --load-mode and picks the right flag. And the rule that went in the journal: after any toolbox refresh, send one real request through every affected route. Version strings lie by omission; requests don't. The refresh also paid for its trouble: +14-29% prefill on the default coder, decode flat.

Honest limits

Mostly single runs on one machine and one quant (the winner's repeat agreed within ~4%); no 262K-context test, no quantized KV with MTP, no long soak. The adopted runtime is experimental and hand-built, unpinned, with no registry copy - and its own README recommends a workaround for a batched-inference bug that none of our checks reproduced. Post-switch verification went through the real router: 49-second load, 62.8 t/s at two concurrent requests, 18/18 concurrency correctness, 4/4 tool calling. The MTP concurrency curve measured on this engine got its own post.

Four claims went in; one engine came out. I'd say the audit's real product isn't the +35-92% - it's the two sentences you can reuse: not reproduced on our metric is not the same as false, and one real request beats any version string...