Speculative decoding is supposed to be output-invariant. Rejected drafts are discarded, accepted tokens must match what the target model alone would have produced, and that guarantee is the entire point of the algorithm. This post reports a measured exception: on two models of the Qwen3.5-MoE lineage, running llama.cpp's native multi-token-prediction (MTP) path on the Strix Halo box, enabling MTP measurably reduced code correctness on hard tasks. For Ornith 1.5 35B A3B the cost was 17% overall and 25% on the hardest task. A paired control on Qwen3.6-35B-A3B replicated the direction at about a third of the magnitude.

The scope claim first, because it matters: this is two models, one lineage, one inference stack, small samples. I am not claiming speculative decoding is broken in general. I am claiming that on these two models it demonstrably is not free, that open llama.cpp bugs make the mechanism plausible, and that the shared lineage may be part of the story. If you run Qwen-lineage MTP under llama.cpp, this is a measure-before-trusting situation.

Where this came from

The Ornith 1.5 benchmark found that the model's native MTP head is a real throughput win on this hardware, +14% to +40% at every concurrency level, once the draft depth is tuned to n=1. That post picked Q8_0 + MTP n=1 as the production config on speed alone and explicitly deferred the quality question. This is the quality question.

The harness

Execution-based scoring only, no LLM judge. Every task ships a real pytest suite; the model's code passes real tests or it does not. Every suite was verified against a hand-written reference solution before any model touched it, which catches bugs in our own tests rather than blaming them on the model. Every config runs identical seeds, so differences are attributable to the config, not to which random generations got drawn. Scoring is partial-credit (fraction of tests passed), which gives materially more resolution than all-or-nothing at small sample sizes. Full raw data kept: every model response, reasoning trace, and pytest output.

The off-the-shelf suites (SWE-bench, BigCodeBench, LiveCodeBench, EvalPlus and friends) were surveyed and passed over: most assume cloud-scale compute or need adapters to point at a local llama-server endpoint, and the question here needed paired seeds against a live server anyway.

Phase one: five easy tasks, at ceiling, telling us nothing

The first suite was five single-function problems with canonical solutions (parse a duration string, fix a buggy LRU cache, aggregate transactions, merge intervals, a token-bucket rate limiter). Three seeds per config across BF16, Q8_0 with and without MTP, Q6_K, Q4_K_M.

Result: near-perfect everywhere. Three configs at 111/111, Q8_0 dropping a single seed on a single task, and MTP exactly quality-neutral: Q8_0 with and without it produced identical outcomes across all 15 seed-runs. I briefly reported that as closing the quality question. It closed nothing. A suite everyone aces cannot separate anything, and the one Q8_0 failure did not even correlate with precision, since the lowest-precision quant was flawless.

The ceiling effect is worth a paragraph of its own because it is the trap in this kind of work. Easy tasks produce clean, confident, useless numbers. If I had stopped there, the conclusion would have been "MTP is quality-neutral," backed by 75 seed-runs, and wrong in the direction that matters.

Phase two: three hard tasks, real signal

The second suite was designed for the model's capability frontier: a dependency-injection container (lifetimes, constructor injection, circular-dependency detection, reverse-order teardown), a two-phase-commit coordinator (the nasty case: a resource that already committed must still roll back when a later commit fails), and a recursive-descent expression parser (precedence, right-associative exponentiation, unary-minus disambiguation). Piloted on BF16 first to confirm real per-seed spread before spending GPU-hours on the grid. Five paired seeds per config, partial credit.

Ornith 1.5, mean fraction of tests passed:

Two findings.

MTP has a real correctness cost on hard tasks. Q8_0 falls from 0.901 to 0.745 with MTP n=1, a 17% relative drop, concentrated on the hardest task: the DI container falls from 0.868 to 0.649, 25%. The easy-task neutrality was real but meaningless. A concrete flavor of the failures: under MTP, one run produced code referencing a _container variable that was never defined anywhere, an error class that appeared in no non-MTP run.

Quantization does not behave monotonically. Q8_0 without MTP scored highest overall, above BF16. Q4_K_M was worst on the DI container (0.592) and best on the parser (0.873); Q6_K mirrored it. No config was uniformly better across the three tasks, which matches the published large-scale quant studies: quantization magnifies a model's existing per-task unevenness rather than degrading cleanly with bit-width.

Why would MTP change outputs at all?

The invariance guarantee should make this result impossible, so I went looking for a mechanism. The evidence ranks like this.

llama.cpp's draft-mtp path has open, reproduced non-determinism bugs on this exact lineage. Two independent reports (#23302, #23335), both on Qwen3.6 MTP models under greedy, seeded, fully deterministic sampling, show the committed token stream changing with the draft setting. Under greedy decoding the invariance should be trivially exact (accept only if the draft equals the argmax), and it is not. The second report shows divergence at n_max=1, the tuned setting used here, not just at deeper drafts. Both issues are open with no public root cause: a reproduced symptom, not a diagnosed line of code, but sufficient on its own to explain wrong-but-plausible tokens landing in generated code.

The Qwen3.5 lineage looks unusually hard to speculate against. A recent paper on self-speculation in hybrid models measured draft/target divergence by architecture: parallel hybrids like Falcon-H1 sit at a total variation distance around 0.3, while the sequential hybrid design (their example is a Qwen3.5 model) measures 0.8, with perplexity 82x more sensitive to attention ablation. Attention is load-bearing and non-redundant in this family. The same paper found acceptance drops sharply on tasks needing long-range dependencies, which maps onto the easy/hard split here: the hard tasks are exactly the ones threading state across a whole file. Different model, same lineage; corroborating, not proof.

The general MTP literature predicts the shape. MTP heads are trained on near-term objectives and their proposal quality decays with distance from the last verified token; reported strengths concentrate on low-entropy, repetitive, structured continuations. Hard multi-constraint code is close to the worst case.

The control: same tasks, same seeds, Qwen3.6-35B-A3B

If the cost were purely architectural (lineage plus llama.cpp bugs), any model in the family should take a comparable hit. If it were purely Ornith's own MTP head being immature, a mature head should show little or none. Qwen3.6-35B-A3B is the same lineage, ships a separately published MTP variant, and is this box's production daily-driver: a natural control. Same three hard tasks, same five paired seeds, same harness.

The direction replicates: 6.8% relative cost overall, with the two-phase-commit task losing a previously perfect score. The magnitude is roughly a third of Ornith's, and the effect is not uniform: the DI container actually improved under MTP. Read at face value, that moves the weight of evidence away from "purely architectural" toward a mix: a small, family-wide cost, consistent with the open llama.cpp bugs firing at some rate, compounded by a larger model-specific cost from Ornith's own younger MTP head. Five seeds by three tasks per arm is a small sample and the mixed per-task signal is real, so treat the attribution as suggestive. What is not ambiguous is that on neither model was MTP free on hard tasks.

What this does and does not claim

Does claim: on these two models, this quant, this stack (llama.cpp draft-mtp on ROCm/gfx1151, though the upstream bug reports come from other backends, so this does not look hardware-specific), MTP n=1 buys throughput at a measurable correctness cost that concentrates on complex, multi-constraint code. And: the easy-task eval that showed perfect neutrality was true and useless at the same time, which is a warning about eval design, not about MTP.

Does not claim: that this generalizes beyond the Qwen3.5-MoE lineage, that the exact magnitudes would survive more seeds, or that speculative decoding with a separate draft model has the same problem (it was not tested here). The shared lineage of both affected models may well be part of the story; two models cannot establish that.

One gap worth naming: everything published on MTP quality that I could find measures perplexity, acceptance rate, or QA benchmarks. I could not find a prior measurement of MTP's effect on execution-graded code correctness, the metric that actually matters for a coding agent. If you know of one, I would like to read it.

Takeaways

• Verify spec-decode invariance on your own stack, against tests that execute. "Speculative decoding never changes outputs" is a property of the algorithm, not of every implementation of it. On this stack it did not hold.
• Eval difficulty is a validity condition. The easy suite produced 75 seed-runs of confident neutrality that phase two overturned. If every config is near-ceiling, the eval is measuring the ceiling.
• Speed and correctness need the same rigor. The throughput bench that picked this config was careful: seeds, paired arms, acceptance-rate diagnosis. It still almost shipped a config that costs a quarter of the hardest task's correctness, because quality was deferred. Bench both before wiring anything into production.

Reproducibility: AMD Ryzen AI MAX+ 395 (Radeon 8060S, gfx1151), 128 GB unified memory, llama.cpp draft-mtp (--spec-type draft-mtp --spec-draft-n-max 1). Ornith 1.5 35B A3B (bartowski GGUFs) on llama.cpp b10530; Qwen3.6-35B-A3B control on the production b10038 container. Custom execution-based harness: pytest-scored tasks self-verified against reference solutions, paired seeds, partial-credit scoring, temp 0.6 per the vendors' agentic-use recommendations. Throughput context in the Ornith post; base stack in the setup guide.

Ornith 1.5 35B A3B is the new agentic coding model from Ornith AI, released on 2026-08-19 with a native multi-token-prediction (MTP) head inherited from its Qwen3.5-MoE base: the model drafts its own future tokens, no separate draft model required. I benchmarked it on the Strix Halo box, and the finding worth leading with is about that head. MTP gives Ornith a real speedup on this hardware, +14% to +40% at every concurrency level tested, but only at draft depth 1. At the commonly recommended depth 3, it runs 8% to 34% slower than no speculation at all. Same model, same head, one flag apart.

That result carries some history. Speculative decoding was 0-for-3 on this hardware before now: ngram drafting was a net negative on real work, and DFlash was unavailable here and recorded 0.000 acceptance on the H100. This is the first win, and the diagnosis took one Prometheus counter and one instrumented run. The per-position acceptance data below explains both halves: why depth 1 pays and why depth 3 cannot.

The model

The details: MIT-licensed, ~35B total parameters, ~3B active MoE, 256K context, published by the team formerly known as DeepReinforce. The training pitch is "self-improvement": an RL pipeline in which the model proposes its own tasks, builds scaffolds for them, and learns from its own rollouts. The release hit the top of Hacker News. The vendor claims 68.5 on Terminal-Bench 2.1 and 79.0 on SWE-bench Verified for the 35B, ahead of its Qwen and Gemma base-model relatives; at least one independent run disputes those numbers, so treat them as vendor figures until the dust settles.

The lineage matters for everything below. Ornith's 35B MoE is a fine-tune of the Qwen3.5 MoE family, which is where its MTP head comes from and, less happily, where its driver problems come from. bartowski GGUFs appeared within a day of release; this bench ran the day after that.

Getting it running without crashing the box

The backstory earns a paragraph. The night started with a different model, Qwen3.8-27B, which crashed the machine twice: hard amdgpu driver hangs on Vulkan RADV, queue evicted, dead kernel log, no clean shutdown. That model family has open, unresolved Vulkan DeviceLost reports on other hardware too. Two hard crashes was my limit. The model is abandoned, the GGUF kept on disk for a future retry.

Ornith is the same architecture family, so it got the treatment the first model should have: research before hardware. Three decisions came out of that. ROCm backend, not Vulkan, based on the family's Vulkan crash reports and a community account of this exact lineage corrupting on Vulkan and being fixed by ROCm. Note the inversion: the DeepSeek post concluded Vulkan was the better backend for that model. Backend choice is per model family, not per box. Second, a newer llama.cpp than the production-frozen ROCm container carries, because Ornith's GGUFs need a recent build; that ran in a separate escape-hatch container, bumped to b10530. Third, the stock chat template has the same bug as its Qwen relatives: it raises an exception on multi-system-message conversations, which means every real agentic client. Patched locally before first use.

The smoke test ran at full BF16, which is more reasonable on this architecture than it sounds. On a sparse MoE, decode cost scales with active-parameter bytes, about 3B here, not the ~70 GB on disk. The whole series has been one long demonstration of that arithmetic. Full-precision decode: 25.6 tok/s. Clean run, tool calling worked, the patched template held on the multi-system-message case.

The default that made it slower

llama.cpp exposes the MTP head through --spec-draft-n-max: how many tokens to draft per round. The guides for this model family recommend depths of 2 to 5. The closest thing to a documented Ornith config I could find used 3, with an external draft model. So the first pass ran n_max=3.

Result: 28% slower single-stream, and negative at every concurrency level, -8% to -34%. The feature that is supposed to be free speedup, as a pure tax.

That sounded backwards, so I instrumented it. llama-server's /metrics endpoint exports per-position draft acceptance counters (spec_decode_num_accepted_tokens_per_pos_total). One single-stream run per setting, BF16:

The head predicts the immediate next token well, around 72-74% acceptance. It collapses at depth: roughly 11% at position 1, 6% at position 2. At n_max=3 every round pays for a three-token verify pass and earns about 1.08 accepted tokens, barely better than not drafting at all, while the wider verify batch costs more. My working hypothesis for the extra cost is that a wider batch touches more distinct experts per pass; that is a hypothesis, not a measurement. n_max=1 keeps the position the head is good at and drops the tax.

One flag. The full sweep confirmed it at every concurrency level: +14% to +40% over no-MTP, where the recommended depth was -8% to -34%.

The quant sweep

Per-stream tok/s, mean of 3 seeds, 4096-token fixed input, 512-token forced output, ROCm, client-side measurement:

One caution on a metric this table could tempt you into: scaling retention, the share of single-stream throughput kept at c=32, runs 8-16% here, and the best retention in the matrix (16.2%) belongs to the slowest config, BF16 at the harmful n=3 default. Starting slow flatters the ratio. Retention only means something next to the absolute numbers.

The single-stream numbers are the headline: 64 tok/s at Q8_0, 78 at Q4_K_M, on an iGPU, for a model whose vendor recipe targets two 80 GB datacenter GPUs. But the more interesting structure is at the other end of the table.

MTP's benefit shrinks with concurrency, and can reverse. At c=32, Q6_K with MTP aggregates 165.7 tok/s against 184.3 without it: the flag that helps at every low concurrency actively hurts there. Q4_K_M's MTP gain essentially vanishes (192.7 vs 189.5). Only BF16 and Q8_0 kept MTP as a net win at every level tested. My working hypothesis: at high concurrency, multi-slot batching already amortizes expert-weight loading across requests, which is much of what speculative drafting buys you at low concurrency, so MTP's fixed per-round overhead eats a shrinking marginal benefit. The crossover arrives sooner the faster the base config already is. Again: hypothesis, consistent with the data, not proven by it.

The benchmark that almost measured the wrong model

Mid-run, a kill -9 between benchmark arms silently failed and the old server kept running. The next arm's launch script health-checked the port, got a 200 from the stale server, and ran a full sweep against the wrong model. The numbers looked completely plausible. Nothing crashed, nothing errored, and the result file would have gone into the matrix if I had not manually cross-checked /v1/models against what was supposedly launched.

This is the second time this series has met a measurement that looked fine and measured nothing (the first was the judge with 100% fake consistency). The driver script is hardened now: refuse to start if any server is already running, verify the reported model ID matches the launched one after the health check, and loop-verify process death instead of trusting one kill and a sleep. We observed a single kill silently failing once; that is enough to stop trusting it forever.

What this bench does not say

It says nothing about output quality. Every number above is throughput. (The follow-up quality evaluation grew into its own post, and it changes the conclusion.) Third-party data puts Q4_K_M around 91% top-1 token agreement with BF16 and Q8_0 around 95.6%, which is why the single-stream crown of Q4_K_M does not make it the pick. And this config is validated, not production: it still needs a real coding-agent quality evaluation, and a long-duration stability run, because the newer ROCm container's runtime line has a separate, unresolved queue-eviction livelock bug that short benchmark sessions would not necessarily surface.

The config that matters for agentic work

The deployment question is what concurrency you actually operate at. In my experience a single serious agentic session is already a parallel workload: a pi-ensemble-style orchestrator fans out specialist subagents, and running one such session comfortably wants more than 8 concurrent streams from the server. That rules out picking the config by its single-stream column.

Read the table at c=8 and c=16 and the answer is Q8_0 with MTP n=1: 22.4 and 12.9 tok/s per stream, best or near-best at every concurrency level, the strongest c=32 aggregate of the nine configs (201.4 tok/s), and the least quality risk short of full precision. That was the pick on throughput alone. The follow-up quality evaluation complicates it, in a way that matters specifically for the hard agentic work this model is positioned for.

Takeaways

• Never ship a community speculative-decoding setting without measuring acceptance on your own hardware and quant. The recommended depth was actively harmful here. The diagnosis cost one instrumented run against a Prometheus counter, and the fix was one flag.
• Draft depth should follow per-position acceptance, not folklore. A head that is 72% right at position 0 and 6% right at position 2 is a one-token drafter, whatever the guides say.
• The scoreboard now reads one win, three losses, and the win required rejecting the default, and then survived only partially: the quality follow-up shows the speed win carries a correctness cost on hard tasks. Speculative decoding on this class of hardware is not free speedup; it is a tuning problem, and speed is only half the measurement.
• Backends are per model family. Vulkan for DeepSeek-V4-Flash, ROCm for this Qwen-lineage MoE, two hard driver crashes as the price of assuming otherwise.

The quality follow-up, in brief

The quality evaluation this post asked for ran the next day, produced a result that overturned its own first phase, and grew into a post of its own. The short version: on an execution-graded hard-task suite, MTP n=1 cost Q8_0 17% of its overall score and 25% on the hardest task, a control run on Qwen3.6-35B-A3B replicated the direction at a third of the magnitude, and open llama.cpp draft-mtp bugs are the leading suspect for the mechanism. The production pick above is therefore a real speed-versus-correctness tradeoff, not a free win. Details, tables, and the mechanism evidence are in the follow-up.

Reproducibility: AMD Ryzen AI MAX+ 395 (Radeon 8060S, gfx1151), 128 GB unified memory, ROCm backend, llama.cpp b10530, bartowski/Ornith-1.5-35B-A3B-GGUF, --spec-draft-n-max 1, chat template patched for multi-system-message conversations. Custom benchmark harness (asyncio/httpx against /v1/completions, client-side TTFT and inter-token deltas, 4096-token fixed input, 512-token forced output via ignore_eos, 3 seeds per concurrency level), because llama-batched-bench reports aggregates only. The setup guide covers the base stack.

Can a 2.6B model hold down a real agent job? Not a demo, not a chat window: the web-research role in my personal assistant stack, where it runs an agent loop with search and page-extract tools and has to come back with a grounded, cited report. I benchmarked it against the 26B-class incumbent under a pre-registered decision rule. It was disqualified. The reason it was disqualified is the interesting part, and it was not hallucination.

The setup

The stack runs on the Strix Halo box: everything local via llama.cpp behind an OpenAI-compatible gateway with llama-swap routing models per role. The assistant itself is a Telegram bot, and its privacy design is the part worth describing. The assistant container lives on an internal-only network with no default route; its only egress is a proxy that allowlists the Telegram API and the local LLM gateway. It cannot browse the web at all. All web work is delegated to a research sidecar: a separate container with real egress, running its own LLM agent loop with web-search and page-extract tools against the same local gateway. Outbound queries pass a PII-rewrite gate before they leave the assistant's zone. The assistant never touches the internet; the sidecar never sees the conversation, only the rewritten research query.

The benched role is the model behind that sidecar's agent loop. The incumbent is gemma-4-26b-a4b, the assistant's main chat model: Google's April Gemma 4 MoE, 25.2B total parameters with 3.8B active, Apache 2.0, and currently ranked first on the FACTS Grounding leaderboard, ahead of several frontier models. The challenger is Liquid AI's LFM2.5-2.6B at Q8_0, all of 3 GB. It was eight days old on bench day, and it is marketed for exactly this job: the release is titled "Deploy Agents Everywhere," the vendor claims it leads instruction-following and tool-use benchmarks against models up to 4x its size, and their launch numbers cite 113 tok/s decode on the Ryzen AI MAX+ 395, the precise chip in this box. If any small model should hold this role, it is this one. (Liquid's own model card does add one honest caveat: not recommended for knowledge-heavy tasks.)

Why audition a 2.6B at all

Not memory. When the sidecar uses the same model as the chat assistant, the weights are already resident and the marginal RAM cost is roughly zero. The real motivation is residency-independence: this box gets switched between large models, and when the chat model is swapped out, research built on it stops working. A pinned 2.6B stays resident through every switch. Research that always works, regardless of what the box is currently serving, is worth something.

And the challenger had a record. Two days after its release it was already running the research role in production, and it ran it for about 2.4 days: 1.46M prompt tokens, 178K generated, zero tool-call parse failures, zero malformed JSON, zero reasoning leaks (llama.cpp's dedicated LFM2 tool-call parser earning its keep), ~71 tok/s decode at Q8_0. Mechanically flawless. I reverted it anyway, because its output in a different role (scheduled monitoring jobs) showed hallucinated facts and protocol violations. That revert was a judgment call, made on vibes, without a controlled measurement.

The DeepSeek post established that fitting on the box is not the bar. This bench exists because running cleanly is not the bar either. The bar is: does the work hold up, and you cannot answer that from an impression.

The bench

The design borrows from the eval literature (RAGAS-style reference-free faithfulness, ALCE-style citation support, FActScore's atomic-fact precision, MT-Bench pairwise judging with position swaps) and rejects the off-the-shelf harnesses, because the "model under test" here is a live production container whose arm-switching is a config mutation with quiescence requirements. Both arms would have become custom script providers anyway; the harness would have added nothing. The result is ~1,100 lines of Python, TDD on the pure logic, and an adversarial critique pass on the methodology draft before any data was collected.

The shape:

• 29 queries mined from real usage: current events, local venues, weather, technical research, comparisons, niche site-specific lookups. Fourteen are time-sensitive. Three are fabrication traps: topics with near-zero web coverage where an honest model reports scarcity and a fabricator invents entities (a term that exists only in a private spec, a plausible-sounding but unpublished benchmark score, a hyper-local news fact). I am keeping the actual trap topics out of this post so they stay reusable.
• 116 live research runs: 29 queries × 2 arms × 2 repeats, arms run back-to-back minutes apart against the same live web, arm order randomized per trial by seeded RNG. This ran against the production stack, with the production cron paused and restored in a finally block, quiescence enforced between trials, and every phase resumable.
• Tier 1, deterministic: extract every cited URL, then resolve it from inside the sidecar's own container, using the same egress and DNS the agent used. NXDOMAIN counts as fabricated unconditionally; a non-existent domain cannot have been read. Template artifacts (example.com, placeholder braces) count as fabricated. A deep-link floor prevents a cite-only-safe-homepages strategy from passing. Protocol compliance per report: non-empty, English, at least one citation, no refusal boilerplate.
• Tier 2, judged: pairwise preference per (query, repeat) by gpt-oss, a model family disjoint from both contestants, since letting the incumbent judge itself invites self-preference bias. The judge sees both reports plus the Tier-1 URL verification results injected as annotations; without that, an LLM judge happily rewards confident citations it cannot check. Every pair is judged twice with positions swapped, and the verdicts must agree or the pair scores as a tie. Separately, an ALCE-style citation-support check: fetch up to three cited pages per report and ask the judge for strict sentence-level entailment, with dead and fabricated citations counted as unsupported so a model cannot improve its support rate by having its bad links excluded.
• A measured noise floor: 15 control pairs, the same model against itself, judged identically. The cross-arm signal has to beat the judge's own same-model noise or the verdict is inconclusive.
• A pre-registered decision rule, fixed before data collection: the challenger is promoted only if it passes a fabrication gate (zero invented domains, dead links ≤5%, plus a manual read of every trap report), a protocol gate (≥95% report compliance), a quality gate (≥45% of decisive judge outcomes, explicitly signed in advance: speed does not buy a clearly worse answer), and a conclusiveness check (noise floor cleared). Statistics at query level: sign test plus Wilson 95% CI on the win share.

The results

Verdict first: LFM2.5-2.6B disqualified on the protocol gate. The quality gate also failed. Incumbent retained.

At the judge: 57 cross-arm pairs, position-consistency 79% against a 60% floor. At query level, 12 decisive wins for the incumbent, 5 for the challenger, 12 undecided. Challenger's decisive win share: 29%, Wilson 95% CI 13-53%, against a pre-registered 45% bar. The control pairs put the judge's same-model noise at a 0.10 deviation from 50%; the cross-arm deviation was 0.21, twice the noise floor, so the verdict is conclusive as a gate decision (directionally clear, though the CI is wide; the rule is a margin-plus-CI rule, not a significance test).

And one number that cuts the other way: citation support. Of the challenger's citations, 51% were supported by the cited page under strict entailment. The incumbent: 33%. The small model's citations, when they existed, verified better.

Finding 1: the failure mode is omission, not fabrication

Every fabrication detector came back clean. Zero invented domains from either model across 116 runs. All trap reports honestly reported scarcity instead of inventing entities. The 2.6B does not make things up at a detectable rate, and neither does the incumbent.

What the 2.6B does instead: 20 of its 58 reports contained zero source URLs. Well-formatted, confidently structured, markdown-clean, and unfalsifiable. The failure concentrates on the harder, sparser queries, and it pairs with length: 473 median words against the incumbent's 189. More words, fewer sources. On one trap query the incumbent cited four real URLs both times; the challenger produced 400-word uncited essays twice.

I find this genuinely more instructive than a hallucination result would have been. Fabrication is the failure everyone tests for, and it did not happen. Omission is quieter. An uncited report does not trip a fact-checker; there is nothing to check. If your acceptance criteria only count invented facts, a model can pass while sliding into prose you cannot verify at all.

The eval literature does treat these as distinct failures: ALCE scores citation recall (uncited claims) separately from citation precision (bad citations), and recent work like Cited but Not Verified measures research agents along exactly this split. What I had not seen stated is how lopsided the split gets at small scale under real agent load: all of this challenger's protocol failure was recall, none of it was precision. There is also a plausible mechanism in the literature: LLMs Get Lost in Multi-Turn Conversation measured an average 39% drop from single-turn to multi-turn performance and attributed most of it to unreliability rather than aptitude. A research agent loop is the multi-turn case by construction. The rule held on turn one and eroded by turn twelve.

Finding 2: decode speed bought nothing

The entire premise of the swap was that a small model makes research faster and lighter. The challenger decodes at ~71 tok/s single-stream on this box; the incumbent is a 26B-class MoE. End-to-end, across 116 runs: 104 seconds mean for the challenger, 107 for the incumbent. The share of runs blowing the 120-second production timeout: 31% versus 32%.

Wall clock in this role is web-search round-trips, page fetches, and tool-call turns. The model's decode speed is a rounding error on top. Regular readers will recognize the shape: this blog spent the summer measuring decode physics, and twice concluded that speed was not the blocker (memory was). Here is the third variant: in an interactive research role, the network is the clock. The tok/s column, the thing the whole small-model case was built on, turned out not to matter at all.

Finding 3: the capability exists, the discipline doesn't

The 51% versus 33% citation-support result deserves its own paragraph, because it complicates the clean story. When the 2.6B cited, it cited more accurately than the model ten times its size. (Both numbers are conservative floors; the entailment check reads raw HTML text windows, and extraction noise hits both arms symmetrically.) The capability to ground claims in sources is present at 2.6B. What is missing is the discipline to keep doing it under multi-step agent pressure, on hard queries, at the end of a long tool loop, with this prompt. That distinction matters for anyone trying to run small models in agent roles: the fix space is prompt hardening and protocol enforcement, not necessarily more parameters. Whether hardening closes the gap is testable, and the bench now exists to test it.

The incumbent, for the record, is not spotless: 81% compliance against its own 95% gate (seven uncited reports, four answers in Finnish despite an English-only contract, on Finland-related topics; Gemma 4 ships with 35+ languages out of the box, and community reports of wrong-language replies from local deployments predate mine), 33% citation support, and the run's only harness timeout. This is the model at the top of a grounding leaderboard, measured in a live role, and the two facts coexist just fine: FACTS Grounding measures faithfulness to provided context in one turn, not citation discipline at the end of a live tool loop. Retention is not an endorsement. It is the pre-registered rule doing its job.

The war story: a judge that measured nothing

The first smoke run produced numbers that looked plausible and were garbage. gpt-oss emits its output in OpenAI's harmony format: an analysis channel (the reasoning) first, then a final channel carrying the actual answer, each wrapped in special-token markup. The final channel arrives last, so a small answer budget gets spent entirely inside the analysis channel and the answer never arrives; OpenAI staff have confirmed exactly this behavior on the model's Hugging Face discussions. My judge harness gave it a 10-token budget. Every response truncated inside the reasoning. The parser then read the truncated markup, found no verdict, and silently defaulted: every judgment parsed as a tie, every entailment as unsupported. Both judging passes truncated identically, so position-consistency read as a perfect 100%.

Ties everywhere, perfect consistency, 0% citation support for both arms. Aggregate metrics that pass a glance and measure nothing. What caught it was not a metric but a smell: a 0% support rate for both arms was implausible, and eyeballing the raw judge outputs showed truncated reasoning markup where verdicts should have been. The fix was a proper final-channel extractor, a 4096-token judge budget, and a rule that truncation now yields "unparseable" rather than a fake verdict.

If you run LLM judges: read the raw outputs before you trust the aggregates, and never let a parse failure default to a valid-looking verdict. My judge had 100% position-consistency while measuring nothing.

What the traps taught

The three fabrication traps were the part of the design I was most curious about. When I reviewed the Feynman research agent, the feature I praised was the verifier that kills dead links and hallucinated references; traps are the adversarial version of the same instinct, aimed at the model instead of the output. They returned a null result that I trust more than a positive one: no invented entities from either model, in any trap report. Both models honestly reported that reliable sources did not exist. Both also got dinged for it, because the compliance rule requires at least one citation per report, and an honest "there is nothing to cite" contains none. The penalty was symmetric, so it did not tilt the verdict, but it is a real flaw in the rule: a compliance metric that cannot distinguish honest abstention from lazy omission penalizes exactly the behavior the traps exist to reward. The next revision of the bench needs an abstention-aware compliance path.

Takeaways

• Bench the role, not the model. The challenger's spec sheet and its mechanical production record both said yes. The role said no. Nothing on a model card measures citation discipline at the end of a twelve-step tool loop on a sparse query.
• A day of rigor beats a month of vibes. The whole thing (literature pass, adversarial methodology review, ~1,100 lines with tests, execution, judgment) was designed, built, and run in one day, and it converted an unfalsifiable hunch into a decision with artifacts. The bench is now a permanent one-command tool; any future small model gets the same audition. I could not find a published example of a home-lab controlled A/B of local models in a live agent role with pre-registered decision rules, which I take not as novelty but as a sign the practice is underused: none of the individual techniques here is original, they are all lifted straight from the eval literature.
• Watch for omission, not just fabrication. At least at this scale and with this prompt, the failure mode of a disciplined small model is not lying. It is confident, well-structured, uncited prose.
• Small-model speed is workload-relative. 71 tok/s of decode bought three seconds of end-to-end latency. In tool-loop roles, the network is the clock. The case for a pinned small model here was never speed; it is residency, and that case survives, waiting on a model (or a prompt) that can keep citing under pressure.

The Taalas post argued that the subagent tier of an agent stack wants a frozen, sufficient, high-volume workhorse, and that small models would increasingly hold those slots. NVIDIA published a whole position paper arguing small models are the future of agentic AI. I still think that is right. This bench adds the qualifier that "sufficient" has to be measured in the role, under load, against a rule you wrote down before you saw the data. My 2.6B candidate was not sufficient yet. The audition process that established that is now the most reusable thing on the box.

AMD announced yesterday that it is acquiring Taalas, a Toronto startup that etches model weights directly into silicon. Terms undisclosed, deal expected to close in Q4. The Register's coverage calls the chips what they are: model-specific integrated circuits. Weights live in a mask-ROM fabric on the die. KV cache and LoRA-class adapters live in on-chip SRAM. No HBM, no advanced packaging, no liquid cooling.

The headline number, from Taalas's February test chip: Llama 3.1 8B at roughly 17,000 tokens per second per user. That is a vendor figure, on an old 8B model, using an aggressive custom 3-bit quantization the company concedes degrades quality (gen 2 moves to standard FP4). Discount it as much as you like. Even at a quarter of the claim it is a different universe from anything else in inference.

I think this is one of the more interesting moves AMD has made against Nvidia's position, and not for the reason the coverage leads with.

Decode is bytes per token. Taalas moves the bytes on-die.

Regular readers have watched this blog measure one lesson from three directions this summer. Decode speed is memory traffic per token. Laguna on Strix Halo: 8B active parameters versus 3B active predicted the performance gap almost exactly. Laguna on a single H100: a 118B MoE out-decodes a dense 27B on the same card because it reads 4.5 GB per token instead of 27. MoE got fast by shrinking the bytes you stream from memory.

Etched weights are the endpoint of that curve. The weights never cross a memory bus at all; they are physically part of the logic. The whole weight-streaming bottleneck, the thing that makes a ~220 GB/s Strix Halo decode at 30 t/s and an HBM-equipped H100 decode at 130, is simply not present in the architecture.

There is a second strategic layer here that I find more interesting than the benchmark. Nvidia's moat is not only CUDA. It is priority access to HBM supply and advanced-packaging capacity, the two binding constraints of the entire AI hardware industry. Taalas chips need neither. They are built on TSMC's mature 6nm process, and per Reuters, only two metal layers are customized per model, so a respin takes about 2 months instead of the ~6 a new processor needs. AMD is not just buying speed. It is buying a way to manufacture inference capacity out of parts of the supply chain nobody is fighting over.

The obvious objection, and the caveat I reached for first: you are stuck with the model you etch.

Model lock-in is already here. We just call it production.

Here is my own datapoint. I coded with GLM 4.7 from November to a few weeks ago. Nine months on one model, in a period when new models shipped monthly. At Taalas's respin cadence, that is four or five etch cycles I would not have used.

That is not laziness. The model is coupled to the harness. Prompts, tool-call conventions, stop behavior, the failure modes you have learned to route around: all of it is tuned against one model's quirks. Swapping models means re-validating the whole workflow, and the re-validation usually costs more than the newer model's marginal quality gain. Anyone running agents in production knows this. The release cadence and the adoption cadence are different clocks, and the press keeps conflating them.

So the supposed fatal flaw of etched silicon, "once deployed you're stuck with that model," is not a new constraint. It is the existing production reality made physical. You are stuck anyway. The question is whether you are stuck at 130 tokens per second or at thousands.

The topology argument goes one step further. My own setup in pi-ensemble uses a frontier-class model for the PM role and faster, cheaper models for the subagent runs. I do not think that shape is temporary; I think it is where agentic production is heading generally. And it maps directly onto the hardware split AMD is reportedly building: flexible GPUs for the tier where you might genuinely want next quarter's smarter model, etched silicon for the high-volume tier. The subagent workhorse is the ideal etch candidate: it dominates token volume, its quality bar is "sufficient," its latency compounds across every fan-out, and its model choice is stable precisely because nobody re-tunes their subagent prompts for fun. The economics of a model-specific chip want a model that is high-volume, quality-stable, and frozen. That is a subagent workhorse, described exactly.

The question that matters for this blog: does any of this reach us?

Everything above is a datacenter story. AMD reportedly plans to pair Taalas accelerators with Instinct GPUs in its Helios racks: prefill on GPUs, decode on etched silicon. Model houses and inference providers, who already keep API models live for a year or more, are the natural first customers. Fine.

But look at the bill of materials. The HC1 is a big die, reticle-sized, yet it sits on a mature 6nm process with no HBM, no advanced packaging, no exotic cooling. Those three absences are most of what makes modern AI accelerators expensive. A single-card product holding a frozen 20B-class model does not have an obvious reason to cost five figures. Whether it could retail in the hundreds of dollars is speculation, mine and nothing more, but nothing in the physics forbids it. A cheap PCIe card that decodes a known-good workhorse model at thousands of tokens per second would slot into a local setup exactly where the subagent tier lives today: PM role on an API or the iGPU, fan-out on the card.

Whether that product ever exists is now entirely AMD's decision, and that is the part of this acquisition I will actually be watching. Taalas as an independent company might eventually have sold silicon to whoever paid. Taalas inside AMD sells where AMD's margins point, and margins point at racks.

Because look at what the local-inference roadmap offers otherwise. Speed on this blog's hardware has come from exactly two kinds of lever this year. Incremental software: speculative decoding, which failed twice on our hardware (ngram net-negative, DFlash broken or unavailable), and KV-cache compression like TurboQuant, which buys capacity, not decode speed. And incremental hardware: each LPDDR generation nudges unified-memory bandwidth up a few tens of percent. Meanwhile the unified-memory thesis I keep defending is a capacity thesis. The Strix Halo box fits models a 5090 cannot hold, at 13 to 30 tokens per second, and nothing on the visible roadmap changes those decode numbers by more than increments.

Etched silicon is the first technology I have seen that could change them by an order of magnitude. If it stays in the racks, the plausible future is a widening split: premium hosted agents running at thousands of tokens per second at premium prices, and local inference keeping its capacity advantage while permanently ceding speed. Local stays the place where things fit; fast becomes something you rent.

I would rather live in the other future, the one with a model card in a PCIe slot. AMD, of all companies, has form here: Strix Halo itself is datacenter-adjacent capability pushed into a consumer box. Whether they do it again with etching is, as far as I can tell, completely open.

The honest caveats

• The 17,000 t/s figure is a vendor benchmark on an 8B model from 2024 in a quality-degrading 3-bit format. No frontier-class chip exists yet; the HC2 (20B parameters per chip, pipeline parallelism beyond that) is due this summer.
• KV cache lives in on-chip SRAM, and SRAM is small. The H100 post's lesson applies unchanged: for long-context agentic work the blocker is session memory, not decode speed. How many 100K-token sessions fit in that SRAM fabric is the spec I want before believing the agentic story end to end.
• The deal has not closed, no roadmap has been announced, and every product claim in this post beyond the HC1's existence is inference or labeled speculation.

The physics is sound, the lock-in objection is weaker than it looks, and the strategic logic is real. What is genuinely undecided is who gets access to the speed. That decision now belongs to AMD, and it will say a lot about what local inference is allowed to become.

Poolside's recipes for Laguna S 2.1 start at an H200 with 141 GB of VRAM. The INT4 checkpoint alone is ~72 GB on disk. An 80 GB H100 appears in no supported configuration, and four days after release, we could not find published numbers for one.

We have exactly one H100 80GB, the shared box from the H100 series. So we tried it. It works: ~130 tok/s single-stream, which is faster than the 27B model we serve in production on the same card. The catch is not speed. It is KV-cache memory: roughly 2 concurrent long-context sessions against the ~256 our production model handles.

This is the companion piece to running Laguna on Strix Halo. Same model, opposite end of the hardware spectrum, same honest-numbers treatment.

The production stack it had to fit into

The context matters because it defines the bar. Our H100 runs 24/7 in UpCloud's fi-hel2 data centre in Helsinki, everything OpenTofu-managed, endpoint behind Caddy with Let's Encrypt. Serving engine is vLLM, v0.24.0 in production. The workload is agentic coding: long contexts, often 100K+, tool calls, tens of concurrent agents, heavily prefix-cache-dependent.

Production model is Qwen3.6-27B-FP8. Since the June posts we moved from the 35B MoE to this dense 27B, and the reason is the same one this whole post turns on: its hybrid-GDN attention gives it an unusually cheap KV cache, ~10 KB per token. That is what lets one card hold ~256 concurrent long-context sessions. Remember that number.

The model under test

Laguna S 2.1, released by Poolside on 2026-07-21. We tested it four days later. 117.6B total parameters, ~8.5B active per token (256 routed experts, top-10 per token plus 1 shared). Open weights under OpenMDW-1.1, commercial use allowed, with BF16/FP8/INT4/NVFP4 variants and DFlash speculative-decode draft models.

The quality claim that makes it interesting: SWE-bench Multilingual 78.5% against Qwen3.6-27B's 71.3%, and Terminal-Bench 70.2 against 59.3. Those are vendor figures, but a +7 point gap on the benchmark closest to our actual workload is worth a Friday evening.

The architecture detail that matters: hybrid attention. Only 12 of 48 layers are global; 36 use a 512-token sliding window, and the KV cache is natively FP8. Effective cost lands around 24 KB per token at long context. Cheap by frontier standards, but 2.4× our production model's, and that ratio decides the ending.

How we tested it without risking production

Two rules: production comes back the same night, and the IaC-managed stack does not change.

Research first. Before touching hardware we ran two multi-agent research workflows, 28 agents total, mining primary sources, GitHub issues and PRs, Reddit, and Hugging Face discussions. Every load-bearing claim was adversarially verified against primary sources. Four-day-old model ecosystems are full of confidently wrong advice, and this step caught some of it (more below).

Then a Friday-night maintenance window, about three hours of endpoint downtime. The production model container was stopped, and Laguna was served from a temporary hot-attached 250 GB scratch volume, in a separate container, on a different port. vLLM v0.25.1 for the trial, required by Laguna's tool-call parsers and the quantized checkpoint. Zero changes to the managed stack. Restoring production was literally docker start vllm.

Community tuning applied: gpu-memory-utilization 0.97, context capped at 128K, PyTorch expandable segments, and Poolside's recommended sampling (temp 0.7, top_p 0.95, top_k 20).

The numbers

All measured, same physical GPU.

Fit. Three configs, all booted cleanly with CUDA graphs on, ~191 s boot each:

Speed, against production Qwen on the identical card:

Why does a 118B model out-run a 27B on the same GPU? Same lesson as the Strix Halo post, from the other direction. Decode is memory-bandwidth-bound. MoE decode only reads the ~8.5B active parameters per token, about 4.5 GB at INT4. The dense 27B reads all ~27 GB every token. Fewer bytes per token, faster decode. Total parameter count is a disk-space number, not a speed number.

One datapoint we have not seen published elsewhere: vLLM's prefix caching works correctly with Laguna's hybrid sliding-window attention. That 0.11 s warm TTFT on a 56K prompt is the proof. This was an open question in the community threads we mined, and for prefix-cache-heavy agentic workloads it is the difference between viable and not.

What didn't work

DFlash shipped broken for this checkpoint. Poolside's own speculative drafter recorded 0.000 draft acceptance, which makes it a pure slowdown: 55 tok/s versus 130 without it. This matches week-1 community reports of drafter/checkpoint mismatches. On Strix Halo, DFlash was unavailable; on the H100 it is available and worse than nothing. The one lever that should help is 0-for-2 across our hardware.

The widely-shared --moe-backend triton advice applies only to the FP8 variant. The INT4 path rejects it. Worth knowing before you copy a config from a thread about a different checkpoint.

War stories

The model was four days old and the ecosystem is raw. Weights were re-uploaded mid-week, drafters shipped broken, and one model-card note was actively misleading. The adversarial source-verification step caught it before it cost us window time.

There was an early panic moment. vLLM's idle-windowed log lines suggested 7 tok/s, and for a few minutes the whole experiment looked like a failure. Actual measurement: 130 tok/s. The log averages throughput over windows that include idle time. Do not trust averaged telemetry; measure.

And a side quest: during setup we discovered the production box had been running for 17 days with no shell access. A first-boot DNS race had silently killed Tailscale enrollment on two consecutive server builds. The endpoint was fine, the monitoring was fine, and nobody had needed to SSH in, so nothing surfaced it. The experiment forced an actual login, which found and fixed it. Sometimes the value of poking production hardware is the poking itself.

Total cost of the experiment: a few euros of GPU time and one Friday-evening maintenance window. Production restored and verified the same night, scratch volume deleted, zero infrastructure drift.

The honest conclusion

No same-card swap. Our workload needs tens of concurrent long-context sessions. After 72 GB of weights, the H100 has ~8 GB left for KV, which buys roughly 2. Speed was never the blocker. Memory is. A model that decodes 50% faster does not help if 254 sessions have nowhere to live.

A real contender on bigger hardware. On 2×H100 with tensor parallelism, or a single H200 or B200, the post-weights KV budget grows to 55-65 GB. At that point Laguna's +7 quality points and +50% single-stream speed make it a serious replacement candidate, not a curiosity.

The next step costs nothing. Speed is now a known quantity on our hardware. Quality on our workload is the open question, and it can be answered with a quality A/B of S 2.1 (there is a free OpenRouter endpoint) against our Qwen on real agent traces. No GPU required.

The vendor's hardware floor was real in the sense that matters for production serving, and beatable in the sense that matters for finding out. One evening, a scratch volume, and a separate container got us first-party numbers for a configuration we could not find published anywhere. That trade is almost always worth it.

Can the biggest, smartest model that physically fits on a Strix Halo box earn a place in the daily rotation? DeepSeek-V4-Flash at IQ2_M is 91 GB, right at the edge of what 128 GB of unified memory allows. It benchmarks like a frontier model: MMLU-Pro 86, GPQA 88, SWE-bench 79. If it ran at usable speed, it would be the best local model this hardware can hold.

So I benchmarked it. It runs. Here is how fast.

Update 2026-08-07: DeepSeek has since shipped V4-Flash-0731, a re-post-trained official release of the model benchmarked here. The speed numbers below should carry over; the quality picture changes. See the update at the end.

The setup

Everything below is measured on my Strix Halo box (AMD Ryzen AI MAX+ 395, 128 GB unified memory, the machine from the setup guide). Model: DeepSeek-V4-Flash, IQ2_M quant, 91 GB on disk. Flags: -fa 1, as always on this machine.

One hard constraint up front: the KV cache must be f16. Quantizing it to q8_0 produces garbage output on this architecture. That is not a tuning preference, it is mandatory, and it costs you memory headroom you do not have much of at 91 GB of weights.

ROCm vs Vulkan

Both backends load and run the model cleanly. No crashes, flash attention works. But they are not equal:

Vulkan wins generation by 33%. For a chat or agentic model, generation speed is the number that matters, so Vulkan is the backend for this model. ROCm only looked competitive on a tiny 64-token prefill; at realistic prompt sizes Vulkan pulls ahead there too.

Generation holds steady at roughly 13 t/s across context. That is below the ~19 t/s figure cited in the research around this model. My guess: that number came from a different quant or from the custom ROCmFPX pipeline, not from stock llama.cpp on gfx1151.

The prefill problem

Generation is slow but usable. Prefill is the real problem.

Prefill drops from 148 t/s at 512 tokens of context to 103 t/s at 4096. It keeps degrading from there. A full-matrix run appeared to hang on the pp16384 test. It had not hung. It was genuinely grinding along at an extrapolated ~50 t/s.

Do the arithmetic on an agentic workload. A 16K-token prompt at ~50 t/s prefill is minutes of waiting before the first output token. Every tool call that re-submits context pays that price again. My suspicion is that DeepSeek-V4's sparse-attention and indexer kernels simply are not well optimized on gfx1151 yet, so the architecture's efficiency tricks turn into overhead here.

Where it lands in the lineup

This makes DeepSeek-V4-Flash the slowest big model on the box, not the crown jewel:

Qwen3.5-122B is twice as fast, half the size on disk, and leaves real memory headroom instead of running at the ceiling. At 91 GB plus f16 KV cache, DeepSeek-V4-Flash leaves almost nothing spare, and running that close to the limit is exactly where this machine's memory-pressure failure modes live.

Honest verdict

Feasible but not practical. That is the whole finding.

DeepSeek-V4-Flash is the highest-quality model that fits on this hardware, and you pay for that quality with 13 t/s generation, prefill that collapses on long prompts, and a memory footprint that crowds out everything else. As a daily workhorse it loses to Qwen3.5-122B on every operational axis. The "big quality" default does not change.

Where it could earn a slot: a rare, load-on-demand "I need the single best answer and I will wait" mode. Short-context reasoning tasks, where its weak prefill does not bite and its benchmark-topping quality does. Wired into llama-swap as an occasional route with Vulkan and f16 KV, evicting everything else first. Not for agentic work. Not for long context. Not for anything interactive.

There is a general lesson in here for local inference on this class of hardware. Fitting is not the bar. A model can load, run cleanly, and pass every smoke test, and still be the wrong choice because the tokens-per-second economics do not work for how you actually use it. Measure generation speed, measure prefill at the prompt sizes your real workloads produce, then decide. The best model you can fit is not automatically the best model you can use.

Update 2026-08-07: V4-Flash-0731 changes the quality math

Five days after this post, DeepSeek released DeepSeek-V4-Flash-0731, the official release that supersedes the preview benchmarked above. Same architecture, same 284B total / 13B active size. All the gains come from re-post-training, which means the speed numbers in this post should transfer unchanged: same weights footprint, same prefill behavior, same ~13 t/s.

What changed is quality, and by a lot if the vendor numbers hold. Per DeepSeek's launch table, 0731 beats the larger V4-Pro Preview on every agentic and coding benchmark they published: Terminal Bench 82.7 vs 72.1, DeepSWE 54.4 vs 12.8, NL2Repo 54.2 vs 38.5. It lands near Opus-class agentic territory while remaining behind Opus 4.8 on every row. The usual caveats apply: vendor-reported, unreleased harness, and BenchLM notes the widely-quoted Terminal-Bench jump compares two different benchmark versions.

For this box, the verdict shifts in one direction only. The practicality problems are architectural, so they stay: prefill still collapses on long prompts, the memory ceiling is still the memory ceiling, and agentic use is still ruled out. But the "load-on-demand, I need the single best answer and I will wait" niche just got meaningfully stronger, because the quality you are waiting for is now higher, especially for coding. Two things to watch before re-testing: Unsloth shipped Q4 and Q8 GGUFs on day one, but sub-100 GB quants in this post's IQ2_M territory were still pending as of early August, and the DSpark speculative-decode module that ships with the model is not yet supported in llama.cpp. When a small quant lands, the numbers above are the baseline to beat.

Laguna-S-2.1 is the agentic-coding model of the moment. poolside released it on 2026-07-21: 118B total parameters, ~8B active MoE, "most capable in its weight class." The hype is all H100s and DGX Sparks.

I run it on an AMD Ryzen AI MAX+ 395 mini-PC. Radeon 8060S integrated GPU, gfx1151, 128 GB of unified LPDDR5X. This is the "can the cheap unified-memory box really run it?" story, with receipts.

Day one, and it just works

llama.cpp merged the laguna architecture two days after release (PR #25165, 2026-07-22). A stock Vulkan build loaded the model within hours of me pulling it.

The serving stack is the same one from the setup guide: llama-swap hot-swapping per-model llama.cpp instances, pre-built gfx1151 toolboxes from kyuz0/amd-strix-halo-toolboxes. Download the 73 GB UD-Q4_K_XL GGUF, drop a conf, add a route, restart. Live.

The quiet miracle is worth stating plainly: a 73 GB model with 256K context across 3 slots, all resident in unified memory on an integrated GPU. No consumer discrete GPU can hold this. You would need multiple cards. That is the whole Strix Halo thesis in one screenshot.

The numbers

Measured on my box, UD-Q4_K_XL, Vulkan RADV, llama.cpp b10118, production flags (-fa 1 --no-mmap -ctk q8_0 -ctv q8_0):

~74 GB resident. Decode barely degrades with context: 30 t/s cold, 27.4 t/s at 16K in. Prefill holds up too, only dropping from 399 to 343 t/s across the same range.

For context, the head-to-head against Qwen3.6-35B-A3B on the same box:

Teaching moment #1: it is all in the active parameters. The ~3× prefill gap and ~1.5× decode gap are not mysterious. 8B active versus 3B active is a 2.7× ratio, and that lands almost exactly on the measured prefill difference. Decode is cushioned by memory bandwidth. Once you internalize "speed is a function of active parameters, not total parameters," every MoE number on this hardware becomes predictable.

The gotchas

Three hard-won bits that make this post worth bookmarking.

1. Vulkan only. The ROCm backend crashes. On gfx1151, HIP flash-attention has no device code for Laguna's head-dim-128 sliding-window layers:

fattn-mma-f16.cuh: no device code compatible with HIP arch 1300

Vulkan RADV handles it fine. Another Strix Halo user confirmed the same crash in the PR thread, so it is not my build.

2. Thinking is load-bearing. Laguna interleaves reasoning with output, and quality craters if you truncate it. The GGUF ships a max_new_tokens default that can cut thinking short. Raise it, and keep enable_thinking on for anything hard.

3. Loading needs a clean GPU. A 73 GB weight load plus staging leaves no room for leftovers. An orphaned model from a prior run caused an ErrorDeviceLost on my first attempt. More on where that road leads below.

The rabbit hole: can we make it faster?

Most posts stop at "it runs." I spent the session trying to make it faster and failed three times. The failures are more instructive than the successes.

ngram speculative decoding: a beautiful net-negative. On a verbatim-echo prompt it hit 94.9 t/s, a 3.2× speedup. On a real code-edit turn it regressed decode to 20.5 t/s, a 32% loss, and on novel reasoning it was neutral. Teaching moment #2: speculative decoding only pays when draft acceptance is high. Real agentic edits diverge from the existing context enough that the draft overhead becomes dead weight. Rejected.

Dropping to Q3: the bandwidth trap. Naive math said 26% smaller weights should mean roughly 25% faster decode. Measured gain: 8.6%. Teaching moment #3: Laguna's decode is not purely bandwidth-bound at 8B active. A fixed per-token cost dominates: attention over a huge KV cache, expert routing, and the always-on shared expert. The Q3 dequant kernel eats some of the savings back on top. Not worth the quality risk. Rejected.

DFlash, poolside's own speculative drafter: the tantalizing one. This is the lever that should work, and it is blocked upstream: the draft GGUF will not load on mainline llama.cpp, and there is an open issue (#25117) measuring it roughly 2× slower on a Strix Halo APU. Community numbers, not mine. So the one real upside lever is both unavailable and possibly counterproductive on this hardware. Watching, not waiting.

Verdict: ~30 t/s is near this silicon's ceiling for an 8B-active MoE with dense attention over a large KV cache. That is a physics-grounded conclusion, not a tuning failure.

The war story

Mid-experiments, I ran a big model load concurrently with a big download. Free RAM hit ~2 GB, and model loads started wedging inside the GPU sub-allocator instead of completing. Then each llama-swap retry piled another one on. The signature:

• 8 processes stuck in uninterruptible D-state, wchan = drm_suballoc_new. Unkillable by any signal. It is a deadlock: memory cannot free because the holders are themselves waiting for memory.
• GTT pinned at 67 GB while llama-swap reported nothing loaded.
• No GPU hang in dmesg. Not a crash, an allocator deadlock. Only a reboot cleared it.

The lesson: on unified-memory boxes, serialize your big I/O. Download fully, then load. Never let two large allocations race. The failure mode is not a clean OOM, it is an unkillable deadlock.

The takeaways

• Yes, a ~€4K integrated-GPU mini-PC runs a frontier-class 118B agentic coder at a genuinely usable ~30 t/s, holding 256K context in unified memory. Two years ago this needed a multi-GPU rig.
• Routing wisdom: use Laguna for long-horizon, terminal-driven agentic work where one better decision saves round-trips. Use a lighter MoE like Qwen3.6-35B for interactive, high-frequency loops. Different tools.
• The unified-memory superpower is capacity, not speed. You will not out-decode a 5090, but you will fit things a 5090 cannot, and for local agentic coding, fitting the model plus huge context beats raw t/s.
• Be honest about the ceiling: prefill ~3× slower than a small MoE, no working speculative decoding, Vulkan-only. All fixable upstream over time.

Reproducibility

Every number above is first-party, measured on my box this week. The DFlash slowdown and poolside's marketing multiples are community and vendor figures, labeled as such.

• Hardware: Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151, RDNA 3.5, 40 CU), 128 GB LPDDR5X-8000 (~220 GB/s real).
• Stack: kyuz0/amd-strix-halo-toolboxes (vulkan-radv), llama.cpp b10118, llama-swap.
• Model: unsloth/Laguna-S-2.1-GGUF UD-Q4_K_XL (73 GB, 3 parts).
• Flags: -ngl 999 -fa 1 --no-mmap -ctk q8_0 -ctv q8_0 --kv-unified --parallel 3 -c 262144 --jinja. Vulkan, not ROCm.
• Bench: llama-batched-bench with -npp 512,4096,16384 -ntg 128 -npl 1.

The setup itself is covered in the Strix Halo setup guide and the gotchas post.

For the last two weeks my feed has been "loop engineering" this, "loop engineering" that. Addy Osmani named it on June 7. Within ten days there were follow-ups from Cobus Greyling, Lushbinary, MindStudio, Louis-François Bouchard, Kilo, Firecrawl, several YouTube videos, an Instagram reel, and a Reddit thread asking whether it is just the next buzzword. Two industry figures got cited everywhere. Boris Cherny, who leads Claude Code at Anthropic: "I don't prompt Claude anymore. I have loops running. My job is to write loops." Peter Steinberger, creator of OpenClaw: "You shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents."

I read enough of these to figure out what was being claimed, and then I had a slightly disorienting realisation. By every definition in those articles, I have been loop engineering for over a year. At one point I actually asked Claude whether we should throw some loop engineering into pi-ensemble. Claude pointed out, with the patience of someone explaining something obvious, that pi-ensemble already is loop engineering.

What the term actually means

The framing that has settled out across the articles is a three-floor stack. Prompt engineering is the ground floor: write a good prompt for a single turn. Harness engineering is the middle floor: design the environment a single agent runs inside (its tools, its context, its rubric). Loop engineering is the top floor: design the system that prompts the harness for you. It runs on a schedule. It spawns sub-agents. It verifies its own output. It decides whether to keep going. The model becomes a subroutine inside your loop, not a chat partner on the other side of a prompt box.

The four-step cycle inside the loop is the same in every article: act, observe, reason, repeat. The articles also converge on the same structural ingredient as the thing that actually makes loops work, which is splitting the maker from the checker. The model that wrote the code is too charitable about its own output. A second agent with a different system prompt, and ideally a different model, catches what the first one talked itself into. Sub-agents in .claude/agents/ (Claude Code) and .codex/agents/ (OpenAI Codex) are the productised primitive for this. Addy Osmani makes it the centrepiece of his post. Boris Cherny describes it as how he actually works. The articles call this the heart of the practice.

The pattern is real. The term is also real, and it does shift the conversation usefully: the leverage point moves from writing prompts to designing the system that writes them.

What I was already running

This is the part that was disorienting.

For the last year my main coding environment has been a forked opencode with a custom multi-agent configuration. The shape: one parent process acting as a project manager, dispatching to specialist children. Developer. Adversarial reviewer. Ops. Explore. Code-review children, one per lens (security, error handling, type safety, performance, architecture, simplicity). The PM holds the workflow state. The children do the work and report back. Nothing in the loop talks to me on a per-turn basis. I give it an issue or a directive; it runs through plan, work, gate, review, until it has produced something to merge, or it has hit something it cannot handle and has to escalate.

This year I rebuilt the whole thing as a clean Pi extension called pi-ensemble. Same architecture, less fork maintenance. The five slash commands cover the cycle Osmani describes almost line for line:

• /start initialises the session: searches memory, indexes the codebase, gathers git/PR/CI state. Discovery.
• /research fans out explore specialists in parallel. Context.
• /plan drafts and classifies a GitHub issue. Intent.
• /work runs the full pipeline: branch, developer, mandatory adversarial gate (up to 3 fix rounds), commit, PR, six-pass code review, CI watch, merge. Act, observe, verify, repeat.
• /review runs the six-lens review on demand against any PR or path.

The maker/checker split that Osmani says is the most useful structural thing in a loop is, in pi-ensemble, two separate gates. The adversarial-developer child gets the diff before any commit and tries to break it. Three rounds of fix-and-retry. If it survives that, the six lens reviewers run in parallel, each pinned to its lens, and the findings get deduplicated and precedence-merged into a verdict. Merge does not happen on a critical verdict without override.

I built none of this because anyone called it loop engineering. I built it because turn-by-turn babysitting of a coding agent on hard tasks does not work, and I needed a system that could grind through real PRs without me holding its hand. The pattern emerged from the problem. I am sitting at three screens, up to 6-7 separate sessions and burning hundreds of millions of tokens a day. It would not be possible if I had to be constantly involved with every decision in every session.

What was actually new about the term

The pattern is older than the term. Geoffrey Huntley's "Ralph" technique (early 2026, before there was a name for any of this) is a one-line shell loop that feeds the same prompt to a fresh agent until a status file says done. The articles correctly cite Ralph as the prior art. My setup is a more structured version of the same idea, with named roles and explicit gates instead of one prompt and a status file. Many other practitioners landed on similar shapes independently. The Anthropic Effective harnesses for long-running agents write-up describes the same primitives. OpenAI's Symphony is a fleet-management layer over the same cycle.

What the term does is consolidate a lot of small individual realisations into one named thing that the field can argue about. That is not nothing. Before the name, you had to spend a paragraph explaining what you were doing. After the name, you can point at the stack and say "this is the loop part" and most people understand. Naming things compresses the discourse, and a compressed discourse moves faster.

The other thing the term does is force the maker/checker question to the front. A lot of the early agentic coding hype was "one big agent that does everything." The loop engineering framing makes it obvious that the interesting design choices are about the structure of the loop, not the capability of the single agent. That is the right place for the leverage to be.

What the articles get wrong

Two things, mostly minor.

First, the articles tend to treat the maker/checker split as something you bolt onto a single-agent setup. In practice, the more useful framing is that the loop is multi-agent by construction. The PM is not an enhanced single agent. It is a different kind of agent, with a different job, that happens to dispatch other agents. Treating the orchestrator as first-class changes the questions you ask about the system.

Second, the cost numbers in the new posts are wild. A six-pass code review at frontier-model rates per PR adds up fast. The H100 in production economics make this more defensible, but the articles tend to gloss the operating envelope. Loop engineering only pays for itself when the loop produces something worth its token budget, which is much harder than getting the loop to run.

What I am taking from this

Mostly that the term is useful enough that I will start using it. "Pi-ensemble is my loop engineering setup" is shorter than what I used to have to say.

The deeper thing is the same observation that comes up every time the field names a pattern that practitioners were already running. The naming compresses the discourse, but it also resets the apparent frontier. Articles dated June 7 onward get framed as "the new wave." Setups that were doing the same thing in March or April look like prior art. There is a slight unfairness in how the credit lands, and a slightly larger unfairness in how the buyer-facing narrative settles ("this just emerged"). Neither is the term's fault. The pattern is older than the name, and the people who needed the pattern figured it out before there was a name for it.

If you are reading the loop engineering articles and thinking "this looks like what I have been doing," you are probably right. The discourse caught up. That is good. Use the name. Cite the framing. And do not be surprised that the actual work, the rubrics inside the loop, the verification step, the taste calibration, did not get easier just because there is a term for the box you put it all in.

Mine are not perfect, by the way. Still tuning. Current state at github.com/randomm/pi-ensemble.

I almost added a seventh role to pi-ensemble this week. The reasoning was plausible enough. When the adversarial gate rejects three rounds in a row with overlapping themes, the loop is signalling that the approach is wrong, not the implementation. A fresh "architect" agent could step back and propose a different frame. The PM (Project Manager) would dispatch it on cap-hit. Clean idea. Easy to specify.

I did the research before writing the prompts. The research said no.

What the data shows

The authoritative source is the MAST paper (Cemri et al., NeurIPS 2025): 1,642 execution traces across 7 popular multi-agent frameworks, 14 distinct failure modes, three categories. The headline finding is that failure rates on state-of-the-art multi-agent systems sit between 41% and 86.7%, and that "performance gains often remain minimal compared to single-agent frameworks or simple baselines like best-of-N sampling."

Two of the 14 failure modes are directly relevant to adding a role:

• Disobey Role Specification: 11.8% of all failures. An agent silently behaves like a different agent. The more roles in the system, the more chances for drift.
• Step Repetition: 13.2% of all failures. The orchestrator loses track of what has already been done. The orchestration prompt grows with each role; the orchestrator's grip on it does not.

Add the broader Inter-Agent Misalignment category (31-32% of failures: conversation reset, task derailment, information withholding, ignoring other agents' input, reasoning-action mismatch) and you have an empirical picture that is not subtle. Coordination is where multi-agent systems actually fail. Not capability. Not model choice. Coordination.

The cost of one more role

The intuition I want to displace is that an additional role costs one role's worth of overhead. It does not. Each new role:

• Expands the orchestrator's decision space on every turn (more dispatch conditions to evaluate, more routing combinations to get right)
• Dilutes instruction density in the orchestrator's prompt (the "lost in the middle" phenomenon kicks in earlier when the prompt is busier)
• Adds a compression event for every handoff (the downstream agent sees the output, not the reasoning behind it)
• Creates a new failure surface (every role drift is a potential bug)

The cost is paid every turn the loop runs. The benefit, in the architect-agent case, would be paid only when the adversarial gate hits a cap with thematic overlap. Low-frequency upside against constant-cost downside. The arithmetic does not work.

What I am doing instead

Doctrine, not a new role. The fix is a few paragraphs in the PM's prompt:

• Watch for the pattern: three adversarial rejections with overlapping themes (not orthogonal local bugs)
• When detected, dispatch the existing @explore specialist with a step-back-framed prompt: "Don't review this diff. Consider whether the whole approach is right. Given the original issue and the recurring finding pattern, is there a fundamentally different way to solve this?"
• Take the result, update the spec, surface to user for approval
• Re-enter from /plan with the revised spec

Zero new roles. Existing roles, different prompts. The fresh-context property I wanted from "architect" is already present in @explore (no awareness of the current diff, no role-bias toward defending it or finding bugs in it). The reframe is the prompt, not the role.

This matches what Augment Code's production Coordinator-Specialist-Verifier pattern does. It also matches the recommendation that comes out of MAST: the structural redesign is "removing agents from the coordination role entirely. Agents execute. A governed state machine coordinates." In pi-ensemble, the PM doctrine is the state machine. Doctrine changes are cheap. Roles are expensive.

This is not the first time the doctrine-not-role move has paid off in pi-ensemble. The developer agent already handles what I call the knee method: an agent learning something mid-work that suggests the spec is wrong, and ploughing on into scope it should not be in. Drew Breunig has written the clearest framing of why this matters. His Spec-Driven Development Triangle treats implementation as a feedback mechanism rather than a one-way pipeline: "the act of writing code improves the spec, and it improves the tests." The doctrine in pi-ensemble's developer prompt is the operational counterpart of that idea. Encounter something unexpected that might change the approach, stop and report to the PM, let the PM decide whether the spec needs updating. The agent is the same role. The behaviour is different because the prompt is.

The general rule

If you are tempted to add a role to a multi-agent system, ask whether the same behaviour can be achieved by a different prompt to an existing role. In my experience, the answer is yes most of the time. The exceptions are rare enough that they should be carefully argued for rather than reached for as the default move.

Less is more, in multi-agent setups as elsewhere. The empirical data agrees, which is the more interesting thing than my taste agreeing.

The previous two posts in this series were benchmarks: first sweep on a dense 27B, then the like-for-like rerun on the same MoE variant Strix Halo runs, with MTP speculative decoding. The benchmarks closed the question of "is it fast enough." This post is about the question that comes next: "what does it actually cost to run, in money, energy, and CO₂."

We have now had the H100 endpoint in real use at Trail Openers for about a week. Several developers using it for coding work, not synthetic load. The energy and footprint numbers are nothing like the "H100 = 700W" reflex would predict, and the marginal cost across real coding traffic lands at a small fraction of what an equivalent volume of frontier-API tokens would have cost. This post walks through both, with the caveats they deserve.

What we are actually running

One H100 80GB SXM in UpCloud's fi-hel2 data centre. UpCloud's published per-hour rate during business hours, lower outside. vLLM 0.21.0 serving Qwen3.6-35B-A3B-FP8 (the MoE variant, 3B active out of 35B total) with MTP speculative decoding, FP8 KV-cache, Marlin MoE backend. Endpoint behind Caddy with HTTPS. Business-hours scheduling: the box comes up in the morning, goes down in the evening, weekends off.

The deployment is OpenTofu, idempotent, one tofu apply from cold. The economic and footprint shape depends on the scheduling. Running 24/7 would cost roughly three times what business-hours-only does, for no additional throughput when nobody is at a keyboard. Scheduled correctly, the monthly cost lands in a tight, predictable range.

Energy: well below the TDP

The reflex when you hear "H100" is "700W card." That number is the datasheet TDP, which assumes a particular workload (dense compute, BF16, GPU saturated). What we are running does not look like that workload.

Measured draw from nvidia-smi integrated over time, across a week of real use:

Three reasons the draw stays low. First, the MoE shape: only ~3B of the 35B parameters activate per token, so the compute per token is a fraction of what a dense 35B would burn. Second, FP8 is roughly 2× more energy-efficient than BF16 for the same arithmetic. Third, vLLM's prefix caching eliminates re-computation across conversational turns, which removes a category of work that would otherwise consume tokens and energy for no marginal benefit.

A live calibration confirmed the meter is unbiased (no methodology bug; the low number is real for this workload). The H100 is not a 700W card in the way most people imagine. It is a 700W card running below 50% utilization for this kind of inference, which is the same as saying it is a ~330W card when it matters.

CO₂: single-digit grams per hour

Helsinki sits on one of the cleanest electricity grids in Europe. Finland's lifecycle factor in May 2026 was 54 gCO₂/kWh per Electricity Maps. Apply that to the measured energy draw with a PUE of 1.2:

The actual week of data confirms the range. Looking at our busiest billed hour (2026-06-15 09:00 UTC, 109M input tokens through the endpoint): 0.317 kWh, 17.1 gCO₂. Most working hours land in the 5-9 gCO₂ range.

Project that to a month of business-hours operation: roughly 2.7-4.6 kg of CO₂. This is comparable to running a household refrigerator for a few weeks, not to anything that should give anyone climate anxiety. The reason is not that AI inference is magically clean. It is that the specific combination of MoE + FP8 + Helsinki grid + business-hours scheduling sits at the favourable end of every variable that determines the footprint.

The UpCloud fi-hel2 facility additionally runs on 100% renewable energy and feeds waste heat into the district heating network (the operator I have been able to identify serves up to ~28,000 homes from this and adjacent facilities). The marginal kilowatt of compute, on top of being clean at the input side, displaces heating fuel at the output side. None of which makes inference free of footprint. It just shifts where the offset comes from.

An important caveat. The CO₂ numbers are estimated, not live-measured. We are using a constant grid factor (54 gCO₂/kWh) and a constant PUE (1.2). Both vary in reality. The energy figures from nvidia-smi are exact (the GPU's total_energy_consumption counter, sampled to a database every five minutes). The carbon translation on top is reasonable but not certified.

Cost: a fixed ceiling instead of a meter

The interesting property of the cost picture is not the absolute number. It is the shape. A rented dedicated GPU costs what it costs whether the team writes one diff or a hundred. There is no surprise bill, no per-token meter spinning faster as the workload scales. For a team that does not yet know how heavily it will use its agents in any given week, that is a structurally different financial risk profile than paying per token to a frontier API.

The marginal cost across real coding traffic comes out well below current frontier-API rates, and well below the published rates for hosted open-weight inference of the same model. The exact ratios depend on the comparison and the load shape, both covered in the next section. The takeaway for budgeting is simpler: instead of an unbounded line item that scales with usage, you get a predictable monthly figure that lands in roughly the same range regardless of how heavily the box gets driven within the working day.

Versus the alternatives

This is where the picture sharpens. Two comparisons that matter:

Versus Anthropic Sonnet 4.6. At our current load shape, our marginal cost is roughly 15-18× cheaper per million tokens than Sonnet's published rates. But the headline ratio understates the difference for the actual shape of agentic coding traffic, which is dramatically input-heavy. The ratio of input to output tokens in our real usage is around 120:1. The agent reads a lot of code and writes a small diff. On real two-hour samples of our actual workload, the same traffic priced on Sonnet would have cost roughly 23-74× more than running it on our own H100, depending on whether the hour was light or heavy. Frontier APIs bleed on input tokens, and agentic coding is the workload where that bleed hurts most.

Versus a hosted open-weight API serving the same Qwen3.6-35B-A3B model. Hosted open-weight inference of this model is priced an order of magnitude below Sonnet, so the gap narrows. In the near-idle state we are roughly at parity. In busy hours, where our utilisation rises and our marginal output cost drops, we are roughly 3.5× cheaper than the hosted alternative. The price advantage of the self-hosted option grows with utilisation. Below a certain steady-state load the hosted API is the right answer; above it, the rented dedicated GPU wins.

This is the part of the picture that surprises people: open-weight models on hosted APIs have already collapsed most of the price gap to running them yourself. The dominant remaining argument for self-hosting is not "it is much cheaper." It is the structural properties: data sovereignty, fixed cost ceiling, predictable monthly accounting, and the ability to integrate the inference endpoint into the same network and trust boundary as the rest of the infrastructure.

The honest caveats

Five things to know before you read these numbers as a guarantee.

This is early operational data. A week of real but light-to-moderate use with some test traffic mixed in. Not a sustained steady-state under heavy 16-agent multi-team load. The benchmarks suggest the operating economics get better at higher utilisation (marginal cost per output token drops), but I cannot show you a month of that yet.

The CO₂ numbers are estimated, not live-measured. Constant 54 gCO₂/kWh Finland factor, constant 1.2 PUE. Both vary in reality; both are reasonable approximations.

MTP acceptance in production is lower than benchmark. The 3.15× single-request uplift in the benchmarks was on --ignore-eos random-token traffic. Real chat workloads see 2.0-2.5× sustained. Already factored into the operational numbers above, just worth saying out loud.

Business-hours scheduling has real ergonomic costs. You cannot run a long-running agent task overnight if the box is down. We have specific workflows that need this (memory consolidation, batch reviews) and we either schedule them to fit the window or accept a 24/7 cost premium for the specific hours we need.

The Trail Openers context is specific. EU jurisdiction, GDPR concerns, the team's physical location matching the data centre, the company's sustainability stance: these are real reasons for us that may or may not be reasons for you. The economic argument generalises better than the locality argument.

What this changes

For Trail Openers, this confirms the architecture decision. The shared H100 in Helsinki is meaningfully cheaper than the alternatives we were comparing against. The monthly cost ceiling is predictable. The footprint is small and on a grid that is cleaner than nearly any hyperscaler default region. And because everything stays in fi-hel2 and on internal endpoints, the data-sovereignty story is clean.

For anyone evaluating a similar setup: the headline economics are real but the durable arguments are structural. A predictable monthly bill instead of an open per-token meter. EU data residency by construction, not by configuration. Clean grid at the input, heat recovery at the output, single-digit kg of CO₂ per month at our scale. The interesting question is not whether self-hosting is cheap. It is whether the structural properties are worth the operational work, and at what team size the answer flips.

For a team of four-to-six developers doing agentic coding, our experience so far is that the answer flipped some time ago.

Telemetry source: nvidia-smi total_energy_consumption (exact GPU counter, driver 595.58.03), sampled to a database every five minutes, then aggregated into hourly usage reports. Cost figures from UpCloud's published per-hour rates. Energy-to-CO₂ translation: constant 54 gCO₂/kWh (Finland lifecycle, Electricity Maps May 2026) × constant 1.2 PUE.