Four coding models on one Strix Halo: pass@1 hides two very different failure modes

The coding-model roster on halo got a proper shootout: four models, two benchmark systems, every number measured on this box. Which model won? That turns out to be the less interesting question. The finding I'd actually lead with: pass@1, the number everyone quotes (the share of problems a model solves on its first try), quietly merges two completely different ways of failing - answering wrong, and never answering at all. And the model ranking changes depending on which failure your workload actually cares about.

The roster

Halo (the AMD Ryzen AI MAX+ 395 box from the setup guide, 128 GB unified memory) currently serves four coding models via llama-swap:

• gemma-4-26b-a4b: Google's Gemma-4 MoE, 25.2B total / 3.8B active, the box's strongest single coding model and the production grounding champion elsewhere in the stack. Now also vision-capable here (more on that another day). A second route runs the same weights with an official Unsloth MTP speculative-decoding sidecar: +68-73% decode, measured.
• qwopus3.6-35b-coder: a community fine-tune of the Qwen3.6-35B-A3B lineage, thinking-off by default, tuned for fast read/edit/test/fix agent loops. The default fan-out model, serving 6 concurrent slots.
• qwen3.8-flash-next-mtp: the Qwen4-preview architecture, now with its native MTP head working via a community-built toolbox. That story earned its own post, coming next.
• north-mini-code: CohereLabs' North-Mini-Code-1.0, a 30B-A3B MoE, smoke-tested this week and not yet in production routing.

Housekeeping note with a number attached: this roster review also decommissioned six models (the whole Nemotron-3 line, Qwen3-Coder-Next, Ornith-1.5, Qwen3.5-122B), freeing 243 GB of disk. Model churn is real; the audition process keeps working in both directions.

Getting comparable numbers: EvalPlus, and the bug that nearly poisoned them

Our own to-bench-style harnesses are great for A/B work but mean nothing to anyone else's hardware. So the roster got scored on EvalPlus: HumanEval+ and MBPP+, the standard, globally-comparable Python coding benchmarks. Anyone can put these numbers next to published ones, which is the whole point of adopting them.

Except the first numbers were garbage, and the way they were garbage is the instructive part. EvalPlus's OpenAI-API backend hardcodes a 768-token generation cap, no flag to raise it. Reasoning models spend their token budget thinking before any code appears - so the harness was silently truncating them mid-thought and scoring the empty result as a wrong answer. The tell was an anomalous number of empty completions. Raising the cap to 8192 (a monkeypatch; there's no flag) fixed most of it, and then revealed a second, genuine finding: some completions stay empty at any budget. On a reproducible fraction of problems, these models reason indefinitely and never answer. Same problems, every run, multiple budgets: a characteristic, not an artifact.

The per-model empty rates at 8192 tokens: qwen3.8-flash-next 12/164 on HumanEval+ and 27/378 on MBPP+ (~7%), north-mini-code 5/164 and 9/378 (~2%), qwopus 1/378, and gemma-4 zero throughout - not because it never overthinks, but because its production config caps reasoning at 1024 tokens, which prevents the failure by construction. Empties are counted as fails in the scores below, which I'd argue is correct: an agent that never answers has failed the task either way.

One more trap for the pile: EvalPlus silently skips re-scoring when a results file already exists. After the patch, the scores did not change until a --i_just_wanna_run flag forced re-evaluation. The series tradition of measuring the wrong thing without an error message continues: your benchmark harness is part of the experiment.

The scores

pass@1, greedy decoding, temperature 0:

Read column by column and the tidy ranking dissolves. HumanEval+ shows Gemma-4 clearly ahead. MBPP+, with shorter and more constrained problems, narrows everything and flips the base ranking outright: North Mini's 0.937 against Gemma-4's 0.939 is a statistical tie at n=378. And the MBPP+ plus column is led by qwen3.8-flash-next, the model with the worst rate of never-finishes-thinking failures in the whole roster.

That last one is the takeaway I'd underline. Flash-Next's failures cluster almost entirely in "never answers", not "answers wrong": when it produces code, the code tends to be correct. Think about what that means for model selection: a model that answers everything with 92% accuracy and a model that answers 92% of problems with near-perfect accuracy carry the same pass@1, and they are completely different tools. One needs review. The other needs a timeout. Raw pass@1 tells you neither, and I'd guess most model-picking decisions out there never split the two.

Honest limits, stated up front rather than in a footnote: EvalPlus is Python-only, single-turn, zero tool use. It does not test the agentic terminal work these models actually do all day. Portable signal, not the whole picture - and the agentic-benchmark gap is now the most obvious hole in this box's evaluation suite.

The realistic speed sweep

For speed, the usual llama-bench synthetic-token corpus was deliberately rejected. Prefill content: an ~855 KB concatenation of the transformers library's actual modeling_*.py source files, sliced at three depths, with prompt caching forced off so repeated depths get no cache discount. Decode task: implement a thread-safe LRU cache with tests. Numbers read from the server's own reported timing fields, not wall-clock. And one deliberate choice worth naming: each model ran its own production route (its real backend, quant, and flags), because the question was "how fast is this model as I actually serve it", not a backend-controlled lab comparison.

Caveat first, because it's a real one: three of the four decode numbers measure reasoning-token throughput, not code-token throughput - the sweep didn't force thinking off, and only Gemma-4's 1024-token reasoning cap left room for actual code inside the test budget. Architecturally, next-token decode cost doesn't depend on what the tokens say, so it's very likely a fair proxy. Flagged, not buried; re-running with thinking forced off is on the list.

The structural finding survives any caveat: Flash-Next's prefill sits at a third to a quarter of the other three, and no amount of MTP decode speedup touches that. Agentic work is prefill-dominated - agents re-read code and re-send long context constantly - so this is the bottleneck that decides real workloads. The receipts are in the benchmark wall-clocks themselves: Flash-Next took ~3.5 hours for HumanEval+ against ~1.5 for the others, and ~8.5 hours for MBPP+ against ~2.5-3, while scoring similar-or-better per problem. Good answers, slowly: the day-one assessment of an immature kernel stack still stands.

Where this leaves the roster

Gemma-4 keeps the crown for single-shot quality, and its MTP route makes it quick too. The fan-out slots stay with qwopus3.6: 64.5 t/s decode, competitive prefill, thinking-off, six parallel slots - the shape agent orchestration wants. North Mini Code earned a real decision: statistically tied with the champion on MBPP+, fastest decode on the box, and not yet wired into production. Whether it displaces anything is exactly what the missing agentic benchmark should decide, and building that benchmark is now the top open thread.

So the roster verdict is provisional in a specific, stated way: single-turn Python is measured, the day job isn't yet. And the question a year into this series keeps getting less like "which model is best" and more like "best at which failure mode, at which speed, in which slot". Early days on that agentic benchmark tho...