Ornith 1.5 35B A3B on Strix Halo: MTP pays off at n=1, not at the default n=3
Ornith 1.5 35B A3B is the new agentic coding model from Ornith AI, released on 2026-08-19 with a native multi-token-prediction (MTP) head inherited from its Qwen3.5-MoE base: the model drafts its own future tokens, no separate draft model required. I benchmarked it on the Strix Halo box, and the finding worth leading with is about that head. MTP gives Ornith a real speedup on this hardware, +14% to +40% at every concurrency level tested, but only at draft depth 1. At the commonly recommended depth 3, it runs 8% to 34% slower than no speculation at all. Same model, same head, one flag apart.
That result carries some history. Speculative decoding was 0-for-3 on this hardware before now: ngram drafting was a net negative on real work, and DFlash was unavailable here and recorded 0.000 acceptance on the H100. This is the first win, and the diagnosis took one Prometheus counter and one instrumented run. The per-position acceptance data below explains both halves: why depth 1 pays and why depth 3 cannot.
The model
The details: MIT-licensed, ~35B total parameters, ~3B active MoE, 256K context, published by the team formerly known as DeepReinforce. The training pitch is "self-improvement": an RL pipeline in which the model proposes its own tasks, builds scaffolds for them, and learns from its own rollouts. The release hit the top of Hacker News. The vendor claims 68.5 on Terminal-Bench 2.1 and 79.0 on SWE-bench Verified for the 35B, ahead of its Qwen and Gemma base-model relatives; at least one independent run disputes those numbers, so treat them as vendor figures until the dust settles.
The lineage matters for everything below. Ornith's 35B MoE is a fine-tune of the Qwen3.5 MoE family, which is where its MTP head comes from and, less happily, where its driver problems come from. bartowski GGUFs appeared within a day of release; this bench ran the day after that.
Getting it running without crashing the box
The backstory earns a paragraph. The night started with a different model, Qwen3.8-27B, which crashed the machine twice: hard amdgpu driver hangs on Vulkan RADV, queue evicted, dead kernel log, no clean shutdown. That model family has open, unresolved Vulkan DeviceLost reports on other hardware too. Two hard crashes was my limit. The model is abandoned, the GGUF kept on disk for a future retry.
Ornith is the same architecture family, so it got the treatment the first model should have: research before hardware. Three decisions came out of that. ROCm backend, not Vulkan, based on the family's Vulkan crash reports and a community account of this exact lineage corrupting on Vulkan and being fixed by ROCm. Note the inversion: the DeepSeek post concluded Vulkan was the better backend for that model. Backend choice is per model family, not per box. Second, a newer llama.cpp than the production-frozen ROCm container carries, because Ornith's GGUFs need a recent build; that ran in a separate escape-hatch container, bumped to b10530. Third, the stock chat template has the same bug as its Qwen relatives: it raises an exception on multi-system-message conversations, which means every real agentic client. Patched locally before first use.
The smoke test ran at full BF16, which is more reasonable on this architecture than it sounds. On a sparse MoE, decode cost scales with active-parameter bytes, about 3B here, not the ~70 GB on disk. The whole series has been one long demonstration of that arithmetic. Full-precision decode: 25.6 tok/s. Clean run, tool calling worked, the patched template held on the multi-system-message case.
The default that made it slower
llama.cpp exposes the MTP head through --spec-draft-n-max: how many tokens to draft per round. The guides for this model family recommend depths of 2 to 5. The closest thing to a documented Ornith config I could find used 3, with an external draft model. So the first pass ran n_max=3.
Result: 28% slower single-stream, and negative at every concurrency level, -8% to -34%. The feature that is supposed to be free speedup, as a pure tax.
That sounded backwards, so I instrumented it. llama-server's /metrics endpoint exports per-position draft acceptance counters (spec_decode_num_accepted_tokens_per_pos_total). One single-stream run per setting, BF16:
| n_max | tok/s | pos-0 accept | pos-1 accept | pos-2 accept |
|---|---|---|---|---|
| 1 | 31.6 | 72.3% | — | — |
| 2 | 23.9 | 63.8% | 12.8% | — |
| 3 | 17.0 | 73.5% | 10.6% | 5.9% |
| no MTP | 25.6 | — | — | — |
The head predicts the immediate next token well, around 72-74% acceptance. It collapses at depth: roughly 11% at position 1, 6% at position 2. At n_max=3 every round pays for a three-token verify pass and earns about 1.08 accepted tokens, barely better than not drafting at all, while the wider verify batch costs more. My working hypothesis for the extra cost is that a wider batch touches more distinct experts per pass; that is a hypothesis, not a measurement. n_max=1 keeps the position the head is good at and drops the tax.
One flag. The full sweep confirmed it at every concurrency level: +14% to +40% over no-MTP, where the recommended depth was -8% to -34%.
The quant sweep
Per-stream tok/s, mean of 3 seeds, 4096-token fixed input, 512-token forced output, ROCm, client-side measurement:
| Config | c=1 | c=4 | c=8 | c=16 | c=32 | agg@32 |
|---|---|---|---|---|---|---|
| BF16, no MTP | 25.6 | 11.0 | 7.5 | 6.3 | 3.7 | 108.3 |
| BF16, MTP n=3 | 18.4 | 9.2 | 6.9 | 4.8 | 3.0 | 83.1 |
| BF16, MTP n=1 | 35.6 | 13.1 | 10.5 | 7.2 | 4.6 | 123.1 |
| Q8_0, no MTP | 47.7 | 28.1 | 20.6 | 11.1 | 6.0 | 178.6 |
| Q8_0, MTP n=1 | 64.1 | 34.9 | 22.4 | 12.9 | 7.2 | 201.4 |
| Q6_K, no MTP | 50.9 | 28.1 | 18.1 | 10.7 | 6.2 | 184.3 |
| Q6_K, MTP n=1 | 71.2 | 30.8 | 23.1 | 11.9 | 5.8 | 165.7 |
| Q4_K_M, no MTP | 59.0 | 30.2 | 18.0 | 11.3 | 6.5 | 189.5 |
| Q4_K_M, MTP n=1 | 77.9 | 34.2 | 22.2 | 13.8 | 6.7 | 192.7 |
One caution on a metric this table could tempt you into: scaling retention, the share of single-stream throughput kept at c=32, runs 8-16% here, and the best retention in the matrix (16.2%) belongs to the slowest config, BF16 at the harmful n=3 default. Starting slow flatters the ratio. Retention only means something next to the absolute numbers.
The single-stream numbers are the headline: 64 tok/s at Q8_0, 78 at Q4_K_M, on an iGPU, for a model whose vendor recipe targets two 80 GB datacenter GPUs. But the more interesting structure is at the other end of the table.
MTP's benefit shrinks with concurrency, and can reverse. At c=32, Q6_K with MTP aggregates 165.7 tok/s against 184.3 without it: the flag that helps at every low concurrency actively hurts there. Q4_K_M's MTP gain essentially vanishes (192.7 vs 189.5). Only BF16 and Q8_0 kept MTP as a net win at every level tested. My working hypothesis: at high concurrency, multi-slot batching already amortizes expert-weight loading across requests, which is much of what speculative drafting buys you at low concurrency, so MTP's fixed per-round overhead eats a shrinking marginal benefit. The crossover arrives sooner the faster the base config already is. Again: hypothesis, consistent with the data, not proven by it.
The benchmark that almost measured the wrong model
Mid-run, a kill -9 between benchmark arms silently failed and the old server kept running. The next arm's launch script health-checked the port, got a 200 from the stale server, and ran a full sweep against the wrong model. The numbers looked completely plausible. Nothing crashed, nothing errored, and the result file would have gone into the matrix if I had not manually cross-checked /v1/models against what was supposedly launched.
This is the second time this series has met a measurement that looked fine and measured nothing (the first was the judge with 100% fake consistency). The driver script is hardened now: refuse to start if any server is already running, verify the reported model ID matches the launched one after the health check, and loop-verify process death instead of trusting one kill and a sleep. We observed a single kill silently failing once; that is enough to stop trusting it forever.
What this bench does not say
It says nothing about output quality. Every number above is throughput. (The follow-up quality evaluation grew into its own post, and it changes the conclusion.) Third-party data puts Q4_K_M around 91% top-1 token agreement with BF16 and Q8_0 around 95.6%, which is why the single-stream crown of Q4_K_M does not make it the pick. And this config is validated, not production: it still needs a real coding-agent quality evaluation, and a long-duration stability run, because the newer ROCm container's runtime line has a separate, unresolved queue-eviction livelock bug that short benchmark sessions would not necessarily surface.
The config that matters for agentic work
The deployment question is what concurrency you actually operate at. In my experience a single serious agentic session is already a parallel workload: a pi-ensemble-style orchestrator fans out specialist subagents, and running one such session comfortably wants more than 8 concurrent streams from the server. That rules out picking the config by its single-stream column.
Read the table at c=8 and c=16 and the answer is Q8_0 with MTP n=1: 22.4 and 12.9 tok/s per stream, best or near-best at every concurrency level, the strongest c=32 aggregate of the nine configs (201.4 tok/s), and the least quality risk short of full precision. That was the pick on throughput alone. The follow-up quality evaluation complicates it, in a way that matters specifically for the hard agentic work this model is positioned for.
Takeaways
- Never ship a community speculative-decoding setting without measuring acceptance on your own hardware and quant. The recommended depth was actively harmful here. The diagnosis cost one instrumented run against a Prometheus counter, and the fix was one flag.
- Draft depth should follow per-position acceptance, not folklore. A head that is 72% right at position 0 and 6% right at position 2 is a one-token drafter, whatever the guides say.
- The scoreboard now reads one win, three losses, and the win required rejecting the default, and then survived only partially: the quality follow-up shows the speed win carries a correctness cost on hard tasks. Speculative decoding on this class of hardware is not free speedup; it is a tuning problem, and speed is only half the measurement.
- Backends are per model family. Vulkan for DeepSeek-V4-Flash, ROCm for this Qwen-lineage MoE, two hard driver crashes as the price of assuming otherwise.
The quality follow-up, in brief
The quality evaluation this post asked for ran the next day, produced a result that overturned its own first phase, and grew into a post of its own. The short version: on an execution-graded hard-task suite, MTP n=1 cost Q8_0 17% of its overall score and 25% on the hardest task, a control run on Qwen3.6-35B-A3B replicated the direction at a third of the magnitude, and open llama.cpp draft-mtp bugs are the leading suspect for the mechanism. The production pick above is therefore a real speed-versus-correctness tradeoff, not a free win. Details, tables, and the mechanism evidence are in the follow-up.
Reproducibility: AMD Ryzen AI MAX+ 395 (Radeon 8060S, gfx1151), 128 GB unified memory, ROCm backend, llama.cpp b10530, bartowski/Ornith-1.5-35B-A3B-GGUF, --spec-draft-n-max 1, chat template patched for multi-system-message conversations. Custom benchmark harness (asyncio/httpx against /v1/completions, client-side TTFT and inter-token deltas, 4096-token fixed input, 512-token forced output via ignore_eos, 3 seeds per concurrency level), because llama-batched-bench reports aggregates only. The setup guide covers the base stack.