AMD announced yesterday that it is acquiring Taalas, a Toronto startup that etches model weights directly into silicon. Terms undisclosed, deal expected to close in Q4. The Register's coverage calls the chips what they are: model-specific integrated circuits. Weights live in a mask-ROM fabric on the die. KV cache and LoRA-class adapters live in on-chip SRAM. No HBM, no advanced packaging, no liquid cooling.

The headline number, from Taalas's February test chip: Llama 3.1 8B at roughly 17,000 tokens per second per user. That is a vendor figure, on an old 8B model, using an aggressive custom 3-bit quantization the company concedes degrades quality (gen 2 moves to standard FP4). Discount it as much as you like. Even at a quarter of the claim it is a different universe from anything else in inference.

I think this is one of the more interesting moves AMD has made against Nvidia's position, and not for the reason the coverage leads with.

Decode is bytes per token. Taalas moves the bytes on-die.

Regular readers have watched this blog measure one lesson from three directions this summer. Decode speed is memory traffic per token. Laguna on Strix Halo: 8B active parameters versus 3B active predicted the performance gap almost exactly. Laguna on a single H100: a 118B MoE out-decodes a dense 27B on the same card because it reads 4.5 GB per token instead of 27. MoE got fast by shrinking the bytes you stream from memory.

Etched weights are the endpoint of that curve. The weights never cross a memory bus at all; they are physically part of the logic. The whole weight-streaming bottleneck, the thing that makes a ~220 GB/s Strix Halo decode at 30 t/s and an HBM-equipped H100 decode at 130, is simply not present in the architecture.

There is a second strategic layer here that I find more interesting than the benchmark. Nvidia's moat is not only CUDA. It is priority access to HBM supply and advanced-packaging capacity, the two binding constraints of the entire AI hardware industry. Taalas chips need neither. They are built on TSMC's mature 6nm process, and per Reuters, only two metal layers are customized per model, so a respin takes about 2 months instead of the ~6 a new processor needs. AMD is not just buying speed. It is buying a way to manufacture inference capacity out of parts of the supply chain nobody is fighting over.

The obvious objection, and the caveat I reached for first: you are stuck with the model you etch.

Model lock-in is already here. We just call it production.

Here is my own datapoint. I coded with GLM 4.7 from November to a few weeks ago. Nine months on one model, in a period when new models shipped monthly. At Taalas's respin cadence, that is four or five etch cycles I would not have used.

That is not laziness. The model is coupled to the harness. Prompts, tool-call conventions, stop behavior, the failure modes you have learned to route around: all of it is tuned against one model's quirks. Swapping models means re-validating the whole workflow, and the re-validation usually costs more than the newer model's marginal quality gain. Anyone running agents in production knows this. The release cadence and the adoption cadence are different clocks, and the press keeps conflating them.

So the supposed fatal flaw of etched silicon, "once deployed you're stuck with that model," is not a new constraint. It is the existing production reality made physical. You are stuck anyway. The question is whether you are stuck at 130 tokens per second or at thousands.

The topology argument goes one step further. My own setup in pi-ensemble uses a frontier-class model for the PM role and faster, cheaper models for the subagent runs. I do not think that shape is temporary; I think it is where agentic production is heading generally. And it maps directly onto the hardware split AMD is reportedly building: flexible GPUs for the tier where you might genuinely want next quarter's smarter model, etched silicon for the high-volume tier. The subagent workhorse is the ideal etch candidate: it dominates token volume, its quality bar is "sufficient," its latency compounds across every fan-out, and its model choice is stable precisely because nobody re-tunes their subagent prompts for fun. The economics of a model-specific chip want a model that is high-volume, quality-stable, and frozen. That is a subagent workhorse, described exactly.

The question that matters for this blog: does any of this reach us?

Everything above is a datacenter story. AMD reportedly plans to pair Taalas accelerators with Instinct GPUs in its Helios racks: prefill on GPUs, decode on etched silicon. Model houses and inference providers, who already keep API models live for a year or more, are the natural first customers. Fine.

But look at the bill of materials. The HC1 is a big die, reticle-sized, yet it sits on a mature 6nm process with no HBM, no advanced packaging, no exotic cooling. Those three absences are most of what makes modern AI accelerators expensive. A single-card product holding a frozen 20B-class model does not have an obvious reason to cost five figures. Whether it could retail in the hundreds of dollars is speculation, mine and nothing more, but nothing in the physics forbids it. A cheap PCIe card that decodes a known-good workhorse model at thousands of tokens per second would slot into a local setup exactly where the subagent tier lives today: PM role on an API or the iGPU, fan-out on the card.

Whether that product ever exists is now entirely AMD's decision, and that is the part of this acquisition I will actually be watching. Taalas as an independent company might eventually have sold silicon to whoever paid. Taalas inside AMD sells where AMD's margins point, and margins point at racks.

Because look at what the local-inference roadmap offers otherwise. Speed on this blog's hardware has come from exactly two kinds of lever this year. Incremental software: speculative decoding, which failed twice on our hardware (ngram net-negative, DFlash broken or unavailable), and KV-cache compression like TurboQuant, which buys capacity, not decode speed. And incremental hardware: each LPDDR generation nudges unified-memory bandwidth up a few tens of percent. Meanwhile the unified-memory thesis I keep defending is a capacity thesis. The Strix Halo box fits models a 5090 cannot hold, at 13 to 30 tokens per second, and nothing on the visible roadmap changes those decode numbers by more than increments.

Etched silicon is the first technology I have seen that could change them by an order of magnitude. If it stays in the racks, the plausible future is a widening split: premium hosted agents running at thousands of tokens per second at premium prices, and local inference keeping its capacity advantage while permanently ceding speed. Local stays the place where things fit; fast becomes something you rent.

I would rather live in the other future, the one with a model card in a PCIe slot. AMD, of all companies, has form here: Strix Halo itself is datacenter-adjacent capability pushed into a consumer box. Whether they do it again with etching is, as far as I can tell, completely open.

The honest caveats

• The 17,000 t/s figure is a vendor benchmark on an 8B model from 2024 in a quality-degrading 3-bit format. No frontier-class chip exists yet; the HC2 (20B parameters per chip, pipeline parallelism beyond that) is due this summer.
• KV cache lives in on-chip SRAM, and SRAM is small. The H100 post's lesson applies unchanged: for long-context agentic work the blocker is session memory, not decode speed. How many 100K-token sessions fit in that SRAM fabric is the spec I want before believing the agentic story end to end.
• The deal has not closed, no roadmap has been announced, and every product claim in this post beyond the HC1's existence is inference or labeled speculation.

The physics is sound, the lock-in objection is weaker than it looks, and the strategic logic is real. What is genuinely undecided is who gets access to the speed. That decision now belongs to AMD, and it will say a lot about what local inference is allowed to become.

Poolside's recipes for Laguna S 2.1 start at an H200 with 141 GB of VRAM. The INT4 checkpoint alone is ~72 GB on disk. An 80 GB H100 appears in no supported configuration, and four days after release, we could not find published numbers for one.

We have exactly one H100 80GB at Trail Openers, the shared box from the H100 series. So we tried it. It works: ~130 tok/s single-stream, which is faster than the 27B model we serve in production on the same card. The catch is not speed. It is KV-cache memory: roughly 2 concurrent long-context sessions against the ~256 our production model handles.

This is the companion piece to running Laguna on Strix Halo. Same model, opposite end of the hardware spectrum, same honest-numbers treatment.

The production stack it had to fit into

The context matters because it defines the bar. Our H100 runs 24/7 in UpCloud's fi-hel2 data centre in Helsinki, everything OpenTofu-managed, endpoint behind Caddy with Let's Encrypt. Serving engine is vLLM, v0.24.0 in production. The workload is agentic coding: long contexts, often 100K+, tool calls, tens of concurrent agents, heavily prefix-cache-dependent.

Production model is Qwen3.6-27B-FP8. Since the June posts we moved from the 35B MoE to this dense 27B, and the reason is the same one this whole post turns on: its hybrid-GDN attention gives it an unusually cheap KV cache, ~10 KB per token. That is what lets one card hold ~256 concurrent long-context sessions. Remember that number.

The model under test

Laguna S 2.1, released by Poolside on 2026-07-21. We tested it four days later. 117.6B total parameters, ~8.5B active per token (256 routed experts, top-10 per token plus 1 shared). Open weights under OpenMDW-1.1, commercial use allowed, with BF16/FP8/INT4/NVFP4 variants and DFlash speculative-decode draft models.

The quality claim that makes it interesting: SWE-bench Multilingual 78.5% against Qwen3.6-27B's 71.3%, and Terminal-Bench 70.2 against 59.3. Those are vendor figures, but a +7 point gap on the benchmark closest to our actual workload is worth a Friday evening.

The architecture detail that matters: hybrid attention. Only 12 of 48 layers are global; 36 use a 512-token sliding window, and the KV cache is natively FP8. Effective cost lands around 24 KB per token at long context. Cheap by frontier standards, but 2.4× our production model's, and that ratio decides the ending.

How we tested it without risking production

Two rules: production comes back the same night, and the IaC-managed stack does not change.

Research first. Before touching hardware we ran two multi-agent research workflows, 28 agents total, mining primary sources, GitHub issues and PRs, Reddit, and Hugging Face discussions. Every load-bearing claim was adversarially verified against primary sources. Four-day-old model ecosystems are full of confidently wrong advice, and this step caught some of it (more below).

Then a Friday-night maintenance window, about three hours of endpoint downtime. The production model container was stopped, and Laguna was served from a temporary hot-attached 250 GB scratch volume, in a separate container, on a different port. vLLM v0.25.1 for the trial, required by Laguna's tool-call parsers and the quantized checkpoint. Zero changes to the managed stack. Restoring production was literally docker start vllm.

Community tuning applied: gpu-memory-utilization 0.97, context capped at 128K, PyTorch expandable segments, and Poolside's recommended sampling (temp 0.7, top_p 0.95, top_k 20).

The numbers

All measured, same physical GPU.

Fit. Three configs, all booted cleanly with CUDA graphs on, ~191 s boot each:

Speed, against production Qwen on the identical card:

Why does a 118B model out-run a 27B on the same GPU? Same lesson as the Strix Halo post, from the other direction. Decode is memory-bandwidth-bound. MoE decode only reads the ~8.5B active parameters per token, about 4.5 GB at INT4. The dense 27B reads all ~27 GB every token. Fewer bytes per token, faster decode. Total parameter count is a disk-space number, not a speed number.

One datapoint we have not seen published elsewhere: vLLM's prefix caching works correctly with Laguna's hybrid sliding-window attention. That 0.11 s warm TTFT on a 56K prompt is the proof. This was an open question in the community threads we mined, and for prefix-cache-heavy agentic workloads it is the difference between viable and not.

What didn't work

DFlash shipped broken for this checkpoint. Poolside's own speculative drafter recorded 0.000 draft acceptance, which makes it a pure slowdown: 55 tok/s versus 130 without it. This matches week-1 community reports of drafter/checkpoint mismatches. On Strix Halo, DFlash was unavailable; on the H100 it is available and worse than nothing. The one lever that should help is 0-for-2 across our hardware.

The widely-shared --moe-backend triton advice applies only to the FP8 variant. The INT4 path rejects it. Worth knowing before you copy a config from a thread about a different checkpoint.

War stories

The model was four days old and the ecosystem is raw. Weights were re-uploaded mid-week, drafters shipped broken, and one model-card note was actively misleading. The adversarial source-verification step caught it before it cost us window time.

There was an early panic moment. vLLM's idle-windowed log lines suggested 7 tok/s, and for a few minutes the whole experiment looked like a failure. Actual measurement: 130 tok/s. The log averages throughput over windows that include idle time. Do not trust averaged telemetry; measure.

And a side quest: during setup we discovered the production box had been running for 17 days with no shell access. A first-boot DNS race had silently killed Tailscale enrollment on two consecutive server builds. The endpoint was fine, the monitoring was fine, and nobody had needed to SSH in, so nothing surfaced it. The experiment forced an actual login, which found and fixed it. Sometimes the value of poking production hardware is the poking itself.

Total cost of the experiment: a few euros of GPU time and one Friday-evening maintenance window. Production restored and verified the same night, scratch volume deleted, zero infrastructure drift.

The honest conclusion

No same-card swap. Our workload needs tens of concurrent long-context sessions. After 72 GB of weights, the H100 has ~8 GB left for KV, which buys roughly 2. Speed was never the blocker. Memory is. A model that decodes 50% faster does not help if 254 sessions have nowhere to live.

A real contender on bigger hardware. On 2×H100 with tensor parallelism, or a single H200 or B200, the post-weights KV budget grows to 55-65 GB. At that point Laguna's +7 quality points and +50% single-stream speed make it a serious replacement candidate, not a curiosity.

The next step costs nothing. Speed is now a known quantity on our hardware. Quality on our workload is the open question, and it can be answered with a quality A/B of S 2.1 (there is a free OpenRouter endpoint) against our Qwen on real agent traces. No GPU required.

The vendor's hardware floor was real in the sense that matters for production serving, and beatable in the sense that matters for finding out. One evening, a scratch volume, and a separate container got us first-party numbers for a configuration we could not find published anywhere. That trade is almost always worth it.

Can the biggest, smartest model that physically fits on a Strix Halo box earn a place in the daily rotation? DeepSeek-V4-Flash at IQ2_M is 91 GB, right at the edge of what 128 GB of unified memory allows. It benchmarks like a frontier model: MMLU-Pro 86, GPQA 88, SWE-bench 79. If it ran at usable speed, it would be the best local model this hardware can hold.

So I benchmarked it. It runs. Here is how fast.

Update 2026-08-07: DeepSeek has since shipped V4-Flash-0731, a re-post-trained official release of the model benchmarked here. The speed numbers below should carry over; the quality picture changes. See the update at the end.

The setup

Everything below is measured on my Strix Halo box (AMD Ryzen AI MAX+ 395, 128 GB unified memory, the machine from the setup guide). Model: DeepSeek-V4-Flash, IQ2_M quant, 91 GB on disk. Flags: -fa 1, as always on this machine.

One hard constraint up front: the KV cache must be f16. Quantizing it to q8_0 produces garbage output on this architecture. That is not a tuning preference, it is mandatory, and it costs you memory headroom you do not have much of at 91 GB of weights.

ROCm vs Vulkan

Both backends load and run the model cleanly. No crashes, flash attention works. But they are not equal:

Vulkan wins generation by 33%. For a chat or agentic model, generation speed is the number that matters, so Vulkan is the backend for this model. ROCm only looked competitive on a tiny 64-token prefill; at realistic prompt sizes Vulkan pulls ahead there too.

Generation holds steady at roughly 13 t/s across context. That is below the ~19 t/s figure cited in the research around this model. My guess: that number came from a different quant or from the custom ROCmFPX pipeline, not from stock llama.cpp on gfx1151.

The prefill problem

Generation is slow but usable. Prefill is the real problem.

Prefill drops from 148 t/s at 512 tokens of context to 103 t/s at 4096. It keeps degrading from there. A full-matrix run appeared to hang on the pp16384 test. It had not hung. It was genuinely grinding along at an extrapolated ~50 t/s.

Do the arithmetic on an agentic workload. A 16K-token prompt at ~50 t/s prefill is minutes of waiting before the first output token. Every tool call that re-submits context pays that price again. My suspicion is that DeepSeek-V4's sparse-attention and indexer kernels simply are not well optimized on gfx1151 yet, so the architecture's efficiency tricks turn into overhead here.

Where it lands in the lineup

This makes DeepSeek-V4-Flash the slowest big model on the box, not the crown jewel:

Qwen3.5-122B is twice as fast, half the size on disk, and leaves real memory headroom instead of running at the ceiling. At 91 GB plus f16 KV cache, DeepSeek-V4-Flash leaves almost nothing spare, and running that close to the limit is exactly where this machine's memory-pressure failure modes live.

Honest verdict

Feasible but not practical. That is the whole finding.

DeepSeek-V4-Flash is the highest-quality model that fits on this hardware, and you pay for that quality with 13 t/s generation, prefill that collapses on long prompts, and a memory footprint that crowds out everything else. As a daily workhorse it loses to Qwen3.5-122B on every operational axis. The "big quality" default does not change.

Where it could earn a slot: a rare, load-on-demand "I need the single best answer and I will wait" mode. Short-context reasoning tasks, where its weak prefill does not bite and its benchmark-topping quality does. Wired into llama-swap as an occasional route with Vulkan and f16 KV, evicting everything else first. Not for agentic work. Not for long context. Not for anything interactive.

There is a general lesson in here for local inference on this class of hardware. Fitting is not the bar. A model can load, run cleanly, and pass every smoke test, and still be the wrong choice because the tokens-per-second economics do not work for how you actually use it. Measure generation speed, measure prefill at the prompt sizes your real workloads produce, then decide. The best model you can fit is not automatically the best model you can use.

Update 2026-08-07: V4-Flash-0731 changes the quality math

Five days after this post, DeepSeek released DeepSeek-V4-Flash-0731, the official release that supersedes the preview benchmarked above. Same architecture, same 284B total / 13B active size. All the gains come from re-post-training, which means the speed numbers in this post should transfer unchanged: same weights footprint, same prefill behavior, same ~13 t/s.

What changed is quality, and by a lot if the vendor numbers hold. Per DeepSeek's launch table, 0731 beats the larger V4-Pro Preview on every agentic and coding benchmark they published: Terminal Bench 82.7 vs 72.1, DeepSWE 54.4 vs 12.8, NL2Repo 54.2 vs 38.5. It lands near Opus-class agentic territory while remaining behind Opus 4.8 on every row. The usual caveats apply: vendor-reported, unreleased harness, and BenchLM notes the widely-quoted Terminal-Bench jump compares two different benchmark versions.

For this box, the verdict shifts in one direction only. The practicality problems are architectural, so they stay: prefill still collapses on long prompts, the memory ceiling is still the memory ceiling, and agentic use is still ruled out. But the "load-on-demand, I need the single best answer and I will wait" niche just got meaningfully stronger, because the quality you are waiting for is now higher, especially for coding. Two things to watch before re-testing: Unsloth shipped Q4 and Q8 GGUFs on day one, but sub-100 GB quants in this post's IQ2_M territory were still pending as of early August, and the DSpark speculative-decode module that ships with the model is not yet supported in llama.cpp. When a small quant lands, the numbers above are the baseline to beat.

Laguna-S-2.1 is the agentic-coding model of the moment. poolside released it on 2026-07-21: 118B total parameters, ~8B active MoE, "most capable in its weight class." The hype is all H100s and DGX Sparks.

I run it on an AMD Ryzen AI MAX+ 395 mini-PC. Radeon 8060S integrated GPU, gfx1151, 128 GB of unified LPDDR5X. This is the "can the cheap unified-memory box really run it?" story, with receipts.

Day one, and it just works

llama.cpp merged the laguna architecture two days after release (PR #25165, 2026-07-22). A stock Vulkan build loaded the model within hours of me pulling it.

The serving stack is the same one from the setup guide: llama-swap hot-swapping per-model llama.cpp instances, pre-built gfx1151 toolboxes from kyuz0/amd-strix-halo-toolboxes. Download the 73 GB UD-Q4_K_XL GGUF, drop a conf, add a route, restart. Live.

The quiet miracle is worth stating plainly: a 73 GB model with 256K context across 3 slots, all resident in unified memory on an integrated GPU. No consumer discrete GPU can hold this. You would need multiple cards. That is the whole Strix Halo thesis in one screenshot.

The numbers

Measured on my box, UD-Q4_K_XL, Vulkan RADV, llama.cpp b10118, production flags (-fa 1 --no-mmap -ctk q8_0 -ctv q8_0):

~74 GB resident. Decode barely degrades with context: 30 t/s cold, 27.4 t/s at 16K in. Prefill holds up too, only dropping from 399 to 343 t/s across the same range.

For context, the head-to-head against Qwen3.6-35B-A3B on the same box:

Teaching moment #1: it is all in the active parameters. The ~3× prefill gap and ~1.5× decode gap are not mysterious. 8B active versus 3B active is a 2.7× ratio, and that lands almost exactly on the measured prefill difference. Decode is cushioned by memory bandwidth. Once you internalize "speed is a function of active parameters, not total parameters," every MoE number on this hardware becomes predictable.

The gotchas

Three hard-won bits that make this post worth bookmarking.

1. Vulkan only. The ROCm backend crashes. On gfx1151, HIP flash-attention has no device code for Laguna's head-dim-128 sliding-window layers:

fattn-mma-f16.cuh: no device code compatible with HIP arch 1300

Vulkan RADV handles it fine. Another Strix Halo user confirmed the same crash in the PR thread, so it is not my build.

2. Thinking is load-bearing. Laguna interleaves reasoning with output, and quality craters if you truncate it. The GGUF ships a max_new_tokens default that can cut thinking short. Raise it, and keep enable_thinking on for anything hard.

3. Loading needs a clean GPU. A 73 GB weight load plus staging leaves no room for leftovers. An orphaned model from a prior run caused an ErrorDeviceLost on my first attempt. More on where that road leads below.

The rabbit hole: can we make it faster?

Most posts stop at "it runs." I spent the session trying to make it faster and failed three times. The failures are more instructive than the successes.

ngram speculative decoding: a beautiful net-negative. On a verbatim-echo prompt it hit 94.9 t/s, a 3.2× speedup. On a real code-edit turn it regressed decode to 20.5 t/s, a 32% loss, and on novel reasoning it was neutral. Teaching moment #2: speculative decoding only pays when draft acceptance is high. Real agentic edits diverge from the existing context enough that the draft overhead becomes dead weight. Rejected.

Dropping to Q3: the bandwidth trap. Naive math said 26% smaller weights should mean roughly 25% faster decode. Measured gain: 8.6%. Teaching moment #3: Laguna's decode is not purely bandwidth-bound at 8B active. A fixed per-token cost dominates: attention over a huge KV cache, expert routing, and the always-on shared expert. The Q3 dequant kernel eats some of the savings back on top. Not worth the quality risk. Rejected.

DFlash, poolside's own speculative drafter: the tantalizing one. This is the lever that should work, and it is blocked upstream: the draft GGUF will not load on mainline llama.cpp, and there is an open issue (#25117) measuring it roughly 2× slower on a Strix Halo APU. Community numbers, not mine. So the one real upside lever is both unavailable and possibly counterproductive on this hardware. Watching, not waiting.

Verdict: ~30 t/s is near this silicon's ceiling for an 8B-active MoE with dense attention over a large KV cache. That is a physics-grounded conclusion, not a tuning failure.

The war story

Mid-experiments, I ran a big model load concurrently with a big download. Free RAM hit ~2 GB, and model loads started wedging inside the GPU sub-allocator instead of completing. Then each llama-swap retry piled another one on. The signature:

• 8 processes stuck in uninterruptible D-state, wchan = drm_suballoc_new. Unkillable by any signal. It is a deadlock: memory cannot free because the holders are themselves waiting for memory.
• GTT pinned at 67 GB while llama-swap reported nothing loaded.
• No GPU hang in dmesg. Not a crash, an allocator deadlock. Only a reboot cleared it.

The lesson: on unified-memory boxes, serialize your big I/O. Download fully, then load. Never let two large allocations race. The failure mode is not a clean OOM, it is an unkillable deadlock.

The takeaways

• Yes, a ~€4K integrated-GPU mini-PC runs a frontier-class 118B agentic coder at a genuinely usable ~30 t/s, holding 256K context in unified memory. Two years ago this needed a multi-GPU rig.
• Routing wisdom: use Laguna for long-horizon, terminal-driven agentic work where one better decision saves round-trips. Use a lighter MoE like Qwen3.6-35B for interactive, high-frequency loops. Different tools.
• The unified-memory superpower is capacity, not speed. You will not out-decode a 5090, but you will fit things a 5090 cannot, and for local agentic coding, fitting the model plus huge context beats raw t/s.
• Be honest about the ceiling: prefill ~3× slower than a small MoE, no working speculative decoding, Vulkan-only. All fixable upstream over time.

Reproducibility

Every number above is first-party, measured on my box this week. The DFlash slowdown and poolside's marketing multiples are community and vendor figures, labeled as such.

• Hardware: Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151, RDNA 3.5, 40 CU), 128 GB LPDDR5X-8000 (~220 GB/s real).
• Stack: kyuz0/amd-strix-halo-toolboxes (vulkan-radv), llama.cpp b10118, llama-swap.
• Model: unsloth/Laguna-S-2.1-GGUF UD-Q4_K_XL (73 GB, 3 parts).
• Flags: -ngl 999 -fa 1 --no-mmap -ctk q8_0 -ctv q8_0 --kv-unified --parallel 3 -c 262144 --jinja. Vulkan, not ROCm.
• Bench: llama-batched-bench with -npp 512,4096,16384 -ntg 128 -npl 1.

The setup itself is covered in the Strix Halo setup guide and the gotchas post.

For the last two weeks my feed has been "loop engineering" this, "loop engineering" that. Addy Osmani named it on June 7. Within ten days there were follow-ups from Cobus Greyling, Lushbinary, MindStudio, Louis-François Bouchard, Kilo, Firecrawl, several YouTube videos, an Instagram reel, and a Reddit thread asking whether it is just the next buzzword. Two industry figures got cited everywhere. Boris Cherny, who leads Claude Code at Anthropic: "I don't prompt Claude anymore. I have loops running. My job is to write loops." Peter Steinberger, creator of OpenClaw: "You shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents."

I read enough of these to figure out what was being claimed, and then I had a slightly disorienting realisation. By every definition in those articles, I have been loop engineering for over a year. At one point I actually asked Claude whether we should throw some loop engineering into pi-ensemble. Claude pointed out, with the patience of someone explaining something obvious, that pi-ensemble already is loop engineering.

What the term actually means

The framing that has settled out across the articles is a three-floor stack. Prompt engineering is the ground floor: write a good prompt for a single turn. Harness engineering is the middle floor: design the environment a single agent runs inside (its tools, its context, its rubric). Loop engineering is the top floor: design the system that prompts the harness for you. It runs on a schedule. It spawns sub-agents. It verifies its own output. It decides whether to keep going. The model becomes a subroutine inside your loop, not a chat partner on the other side of a prompt box.

The four-step cycle inside the loop is the same in every article: act, observe, reason, repeat. The articles also converge on the same structural ingredient as the thing that actually makes loops work, which is splitting the maker from the checker. The model that wrote the code is too charitable about its own output. A second agent with a different system prompt, and ideally a different model, catches what the first one talked itself into. Sub-agents in .claude/agents/ (Claude Code) and .codex/agents/ (OpenAI Codex) are the productised primitive for this. Addy Osmani makes it the centrepiece of his post. Boris Cherny describes it as how he actually works. The articles call this the heart of the practice.

The pattern is real. The term is also real, and it does shift the conversation usefully: the leverage point moves from writing prompts to designing the system that writes them.

What I was already running

This is the part that was disorienting.

For the last year my main coding environment has been a forked opencode with a custom multi-agent configuration. The shape: one parent process acting as a project manager, dispatching to specialist children. Developer. Adversarial reviewer. Ops. Explore. Code-review children, one per lens (security, error handling, type safety, performance, architecture, simplicity). The PM holds the workflow state. The children do the work and report back. Nothing in the loop talks to me on a per-turn basis. I give it an issue or a directive; it runs through plan, work, gate, review, until it has produced something to merge, or it has hit something it cannot handle and has to escalate.

This year I rebuilt the whole thing as a clean Pi extension called pi-ensemble. Same architecture, less fork maintenance. The five slash commands cover the cycle Osmani describes almost line for line:

• /start initialises the session: searches memory, indexes the codebase, gathers git/PR/CI state. Discovery.
• /research fans out explore specialists in parallel. Context.
• /plan drafts and classifies a GitHub issue. Intent.
• /work runs the full pipeline: branch, developer, mandatory adversarial gate (up to 3 fix rounds), commit, PR, six-pass code review, CI watch, merge. Act, observe, verify, repeat.
• /review runs the six-lens review on demand against any PR or path.

The maker/checker split that Osmani says is the most useful structural thing in a loop is, in pi-ensemble, two separate gates. The adversarial-developer child gets the diff before any commit and tries to break it. Three rounds of fix-and-retry. If it survives that, the six lens reviewers run in parallel, each pinned to its lens, and the findings get deduplicated and precedence-merged into a verdict. Merge does not happen on a critical verdict without override.

I built none of this because anyone called it loop engineering. I built it because turn-by-turn babysitting of a coding agent on hard tasks does not work, and I needed a system that could grind through real PRs without me holding its hand. The pattern emerged from the problem. I am sitting at three screens, up to 6-7 separate sessions and burning hundreds of millions of tokens a day. It would not be possible if I had to be constantly involved with every decision in every session.

What was actually new about the term

The pattern is older than the term. Geoffrey Huntley's "Ralph" technique (early 2026, before there was a name for any of this) is a one-line shell loop that feeds the same prompt to a fresh agent until a status file says done. The articles correctly cite Ralph as the prior art. My setup is a more structured version of the same idea, with named roles and explicit gates instead of one prompt and a status file. Many other practitioners landed on similar shapes independently. The Anthropic Effective harnesses for long-running agents write-up describes the same primitives. OpenAI's Symphony is a fleet-management layer over the same cycle.

What the term does is consolidate a lot of small individual realisations into one named thing that the field can argue about. That is not nothing. Before the name, you had to spend a paragraph explaining what you were doing. After the name, you can point at the stack and say "this is the loop part" and most people understand. Naming things compresses the discourse, and a compressed discourse moves faster.

The other thing the term does is force the maker/checker question to the front. A lot of the early agentic coding hype was "one big agent that does everything." The loop engineering framing makes it obvious that the interesting design choices are about the structure of the loop, not the capability of the single agent. That is the right place for the leverage to be.

What the articles get wrong

Two things, mostly minor.

First, the articles tend to treat the maker/checker split as something you bolt onto a single-agent setup. In practice, the more useful framing is that the loop is multi-agent by construction. The PM is not an enhanced single agent. It is a different kind of agent, with a different job, that happens to dispatch other agents. Treating the orchestrator as first-class changes the questions you ask about the system.

Second, the cost numbers in the new posts are wild. A six-pass code review at frontier-model rates per PR adds up fast. The H100 in production economics make this more defensible, but the articles tend to gloss the operating envelope. Loop engineering only pays for itself when the loop produces something worth its token budget, which is much harder than getting the loop to run.

What I am taking from this

Mostly that the term is useful enough that I will start using it. "Pi-ensemble is my loop engineering setup" is shorter than what I used to have to say.

The deeper thing is the same observation that comes up every time the field names a pattern that practitioners were already running. The naming compresses the discourse, but it also resets the apparent frontier. Articles dated June 7 onward get framed as "the new wave." Setups that were doing the same thing in March or April look like prior art. There is a slight unfairness in how the credit lands, and a slightly larger unfairness in how the buyer-facing narrative settles ("this just emerged"). Neither is the term's fault. The pattern is older than the name, and the people who needed the pattern figured it out before there was a name for it.

If you are reading the loop engineering articles and thinking "this looks like what I have been doing," you are probably right. The discourse caught up. That is good. Use the name. Cite the framing. And do not be surprised that the actual work, the rubrics inside the loop, the verification step, the taste calibration, did not get easier just because there is a term for the box you put it all in.

Mine are not perfect, by the way. Still tuning. Current state at github.com/randomm/pi-ensemble.

I almost added a seventh role to pi-ensemble this week. The reasoning was plausible enough. When the adversarial gate rejects three rounds in a row with overlapping themes, the loop is signalling that the approach is wrong, not the implementation. A fresh "architect" agent could step back and propose a different frame. The PM (Project Manager) would dispatch it on cap-hit. Clean idea. Easy to specify.

I did the research before writing the prompts. The research said no.

What the data shows

The authoritative source is the MAST paper (Cemri et al., NeurIPS 2025): 1,642 execution traces across 7 popular multi-agent frameworks, 14 distinct failure modes, three categories. The headline finding is that failure rates on state-of-the-art multi-agent systems sit between 41% and 86.7%, and that "performance gains often remain minimal compared to single-agent frameworks or simple baselines like best-of-N sampling."

Two of the 14 failure modes are directly relevant to adding a role:

• Disobey Role Specification: 11.8% of all failures. An agent silently behaves like a different agent. The more roles in the system, the more chances for drift.
• Step Repetition: 13.2% of all failures. The orchestrator loses track of what has already been done. The orchestration prompt grows with each role; the orchestrator's grip on it does not.

Add the broader Inter-Agent Misalignment category (31-32% of failures: conversation reset, task derailment, information withholding, ignoring other agents' input, reasoning-action mismatch) and you have an empirical picture that is not subtle. Coordination is where multi-agent systems actually fail. Not capability. Not model choice. Coordination.

The cost of one more role

The intuition I want to displace is that an additional role costs one role's worth of overhead. It does not. Each new role:

• Expands the orchestrator's decision space on every turn (more dispatch conditions to evaluate, more routing combinations to get right)
• Dilutes instruction density in the orchestrator's prompt (the "lost in the middle" phenomenon kicks in earlier when the prompt is busier)
• Adds a compression event for every handoff (the downstream agent sees the output, not the reasoning behind it)
• Creates a new failure surface (every role drift is a potential bug)

The cost is paid every turn the loop runs. The benefit, in the architect-agent case, would be paid only when the adversarial gate hits a cap with thematic overlap. Low-frequency upside against constant-cost downside. The arithmetic does not work.

What I am doing instead

Doctrine, not a new role. The fix is a few paragraphs in the PM's prompt:

• Watch for the pattern: three adversarial rejections with overlapping themes (not orthogonal local bugs)
• When detected, dispatch the existing @explore specialist with a step-back-framed prompt: "Don't review this diff. Consider whether the whole approach is right. Given the original issue and the recurring finding pattern, is there a fundamentally different way to solve this?"
• Take the result, update the spec, surface to user for approval
• Re-enter from /plan with the revised spec

Zero new roles. Existing roles, different prompts. The fresh-context property I wanted from "architect" is already present in @explore (no awareness of the current diff, no role-bias toward defending it or finding bugs in it). The reframe is the prompt, not the role.

This matches what Augment Code's production Coordinator-Specialist-Verifier pattern does. It also matches the recommendation that comes out of MAST: the structural redesign is "removing agents from the coordination role entirely. Agents execute. A governed state machine coordinates." In pi-ensemble, the PM doctrine is the state machine. Doctrine changes are cheap. Roles are expensive.

This is not the first time the doctrine-not-role move has paid off in pi-ensemble. The developer agent already handles what I call the knee method: an agent learning something mid-work that suggests the spec is wrong, and ploughing on into scope it should not be in. Drew Breunig has written the clearest framing of why this matters. His Spec-Driven Development Triangle treats implementation as a feedback mechanism rather than a one-way pipeline: "the act of writing code improves the spec, and it improves the tests." The doctrine in pi-ensemble's developer prompt is the operational counterpart of that idea. Encounter something unexpected that might change the approach, stop and report to the PM, let the PM decide whether the spec needs updating. The agent is the same role. The behaviour is different because the prompt is.

The general rule

If you are tempted to add a role to a multi-agent system, ask whether the same behaviour can be achieved by a different prompt to an existing role. In my experience, the answer is yes most of the time. The exceptions are rare enough that they should be carefully argued for rather than reached for as the default move.

Less is more, in multi-agent setups as elsewhere. The empirical data agrees, which is the more interesting thing than my taste agreeing.

The previous two posts in this series were benchmarks: first sweep on a dense 27B, then the like-for-like rerun on the same MoE variant Strix Halo runs, with MTP speculative decoding. The benchmarks closed the question of "is it fast enough." This post is about the question that comes next: "what does it actually cost to run, in money, energy, and CO₂."

We have now had the H100 endpoint in real use at Trail Openers for about a week. Several developers using it for coding work, not synthetic load. The energy and footprint numbers are nothing like the "H100 = 700W" reflex would predict, and the marginal cost across real coding traffic lands at a small fraction of what an equivalent volume of frontier-API tokens would have cost. This post walks through both, with the caveats they deserve.

What we are actually running

One H100 80GB SXM in UpCloud's fi-hel2 data centre. UpCloud's published per-hour rate during business hours, lower outside. vLLM 0.21.0 serving Qwen3.6-35B-A3B-FP8 (the MoE variant, 3B active out of 35B total) with MTP speculative decoding, FP8 KV-cache, Marlin MoE backend. Endpoint behind Caddy with HTTPS. Business-hours scheduling: the box comes up in the morning, goes down in the evening, weekends off.

The deployment is OpenTofu, idempotent, one tofu apply from cold. The economic and footprint shape depends on the scheduling. Running 24/7 would cost roughly three times what business-hours-only does, for no additional throughput when nobody is at a keyboard. Scheduled correctly, the monthly cost lands in a tight, predictable range.

Energy: well below the TDP

The reflex when you hear "H100" is "700W card." That number is the datasheet TDP, which assumes a particular workload (dense compute, BF16, GPU saturated). What we are running does not look like that workload.

Measured draw from nvidia-smi integrated over time, across a week of real use:

Three reasons the draw stays low. First, the MoE shape: only ~3B of the 35B parameters activate per token, so the compute per token is a fraction of what a dense 35B would burn. Second, FP8 is roughly 2× more energy-efficient than BF16 for the same arithmetic. Third, vLLM's prefix caching eliminates re-computation across conversational turns, which removes a category of work that would otherwise consume tokens and energy for no marginal benefit.

A live calibration confirmed the meter is unbiased (no methodology bug; the low number is real for this workload). The H100 is not a 700W card in the way most people imagine. It is a 700W card running below 50% utilization for this kind of inference, which is the same as saying it is a ~330W card when it matters.

CO₂: single-digit grams per hour

Helsinki sits on one of the cleanest electricity grids in Europe. Finland's lifecycle factor in May 2026 was 54 gCO₂/kWh per Electricity Maps. Apply that to the measured energy draw with a PUE of 1.2:

The actual week of data confirms the range. Looking at our busiest billed hour (2026-06-15 09:00 UTC, 109M input tokens through the endpoint): 0.317 kWh, 17.1 gCO₂. Most working hours land in the 5-9 gCO₂ range.

Project that to a month of business-hours operation: roughly 2.7-4.6 kg of CO₂. This is comparable to running a household refrigerator for a few weeks, not to anything that should give anyone climate anxiety. The reason is not that AI inference is magically clean. It is that the specific combination of MoE + FP8 + Helsinki grid + business-hours scheduling sits at the favourable end of every variable that determines the footprint.

The UpCloud fi-hel2 facility additionally runs on 100% renewable energy and feeds waste heat into the district heating network (the operator I have been able to identify serves up to ~28,000 homes from this and adjacent facilities). The marginal kilowatt of compute, on top of being clean at the input side, displaces heating fuel at the output side. None of which makes inference free of footprint. It just shifts where the offset comes from.

An important caveat. The CO₂ numbers are estimated, not live-measured. We are using a constant grid factor (54 gCO₂/kWh) and a constant PUE (1.2). Both vary in reality. The energy figures from nvidia-smi are exact (the GPU's total_energy_consumption counter, sampled to a database every five minutes). The carbon translation on top is reasonable but not certified.

Cost: a fixed ceiling instead of a meter

The interesting property of the cost picture is not the absolute number. It is the shape. A rented dedicated GPU costs what it costs whether the team writes one diff or a hundred. There is no surprise bill, no per-token meter spinning faster as the workload scales. For a team that does not yet know how heavily it will use its agents in any given week, that is a structurally different financial risk profile than paying per token to a frontier API.

The marginal cost across real coding traffic comes out well below current frontier-API rates, and well below the published rates for hosted open-weight inference of the same model. The exact ratios depend on the comparison and the load shape, both covered in the next section. The takeaway for budgeting is simpler: instead of an unbounded line item that scales with usage, you get a predictable monthly figure that lands in roughly the same range regardless of how heavily the box gets driven within the working day.

Versus the alternatives

This is where the picture sharpens. Two comparisons that matter:

Versus Anthropic Sonnet 4.6. At our current load shape, our marginal cost is roughly 15-18× cheaper per million tokens than Sonnet's published rates. But the headline ratio understates the difference for the actual shape of agentic coding traffic, which is dramatically input-heavy. The ratio of input to output tokens in our real usage is around 120:1. The agent reads a lot of code and writes a small diff. On real two-hour samples of our actual workload, the same traffic priced on Sonnet would have cost roughly 23-74× more than running it on our own H100, depending on whether the hour was light or heavy. Frontier APIs bleed on input tokens, and agentic coding is the workload where that bleed hurts most.

Versus a hosted open-weight API serving the same Qwen3.6-35B-A3B model. Hosted open-weight inference of this model is priced an order of magnitude below Sonnet, so the gap narrows. In the near-idle state we are roughly at parity. In busy hours, where our utilisation rises and our marginal output cost drops, we are roughly 3.5× cheaper than the hosted alternative. The price advantage of the self-hosted option grows with utilisation. Below a certain steady-state load the hosted API is the right answer; above it, the rented dedicated GPU wins.

This is the part of the picture that surprises people: open-weight models on hosted APIs have already collapsed most of the price gap to running them yourself. The dominant remaining argument for self-hosting is not "it is much cheaper." It is the structural properties: data sovereignty, fixed cost ceiling, predictable monthly accounting, and the ability to integrate the inference endpoint into the same network and trust boundary as the rest of the infrastructure.

The honest caveats

Five things to know before you read these numbers as a guarantee.

This is early operational data. A week of real but light-to-moderate use with some test traffic mixed in. Not a sustained steady-state under heavy 16-agent multi-team load. The benchmarks suggest the operating economics get better at higher utilisation (marginal cost per output token drops), but I cannot show you a month of that yet.

The CO₂ numbers are estimated, not live-measured. Constant 54 gCO₂/kWh Finland factor, constant 1.2 PUE. Both vary in reality; both are reasonable approximations.

MTP acceptance in production is lower than benchmark. The 3.15× single-request uplift in the benchmarks was on --ignore-eos random-token traffic. Real chat workloads see 2.0-2.5× sustained. Already factored into the operational numbers above, just worth saying out loud.

Business-hours scheduling has real ergonomic costs. You cannot run a long-running agent task overnight if the box is down. We have specific workflows that need this (memory consolidation, batch reviews) and we either schedule them to fit the window or accept a 24/7 cost premium for the specific hours we need.

The Trail Openers context is specific. EU jurisdiction, GDPR concerns, the team's physical location matching the data centre, the company's sustainability stance: these are real reasons for us that may or may not be reasons for you. The economic argument generalises better than the locality argument.

What this changes

For Trail Openers, this confirms the architecture decision. The shared H100 in Helsinki is meaningfully cheaper than the alternatives we were comparing against. The monthly cost ceiling is predictable. The footprint is small and on a grid that is cleaner than nearly any hyperscaler default region. And because everything stays in fi-hel2 and on internal endpoints, the data-sovereignty story is clean.

For anyone evaluating a similar setup: the headline economics are real but the durable arguments are structural. A predictable monthly bill instead of an open per-token meter. EU data residency by construction, not by configuration. Clean grid at the input, heat recovery at the output, single-digit kg of CO₂ per month at our scale. The interesting question is not whether self-hosting is cheap. It is whether the structural properties are worth the operational work, and at what team size the answer flips.

For a team of four-to-six developers doing agentic coding, our experience so far is that the answer flipped some time ago.

Telemetry source: nvidia-smi total_energy_consumption (exact GPU counter, driver 595.58.03), sampled to a database every five minutes, then aggregated into hourly usage reports. Cost figures from UpCloud's published per-hour rates. Energy-to-CO₂ translation: constant 54 gCO₂/kWh (Finland lifecycle, Electricity Maps May 2026) × constant 1.2 PUE.

I have been benchmarking Qwen3.6 on an UpCloud H100 80GB SXM, comparing it against the numbers I have been running on my Strix Halo box. The first sweep, four days ago, used Qwen3.6-27B-FP8 (dense) on the H100 under vLLM 0.21, and my established Qwen3.6-35B MoE Q8 numbers on Strix Halo under llama.cpp. The directional answer it gave me was "single-user generation is closer than you would think; prefill and concurrency win." That answer turns out to have been half-right and half-misleading. I ran the like-for-like comparison today and want to correct the picture.

The context for this work has not changed. At Trail Openers we are moving toward more environmentally sustainable LLM infrastructure for our internal tooling, and the H100 setup we are building will be shared by several developers. UpCloud was chosen for three reasons that compound. First, their data centres run majority on renewable energy and their scope 1+2 emissions are compensated. Second, the fi-hel2 data centre is in Helsinki, the same metropolitan area as Trail Openers, which means the whole team gets single-digit-millisecond latency to the endpoint. Third, EU jurisdiction and GDPR-native data handling matter for the work we do; the prompts, the diffs, the codebases, none of that leaves the EU.

What the first sweep got wrong

The first benchmark ran a dense 27B on the H100 against an MoE on Strix Halo. That comparison was fair on weight footprint (28-34 GB class) but unfair on architectural fit. Dense and MoE behave very differently per-token, and the model choice on each side biased the result in opposite directions.

The summary numbers from that first sweep:

The 1.4× looked like good news for Strix Halo. It is not. It is the artefact of two model choices, not a real architectural finding. Strix Halo was running the MoE variant that suits its ~225 GB/s bandwidth (only the active experts have to be streamed per token). The H100 was running a dense model where every parameter is touched on every forward pass. That is not a fair fight in either direction.

The fair comparison is running the same MoE variant on both, with each platform's best serving stack and best decoding tricks. So I did that.

The like-for-like sweep

Yesterday I redeployed the H100 with Qwen3.6-35B-A3B-FP8 (the same 3B-active / 35B-total MoE Strix Halo runs) plus MTP speculative decoding turned on (--speculative-config '{"method":"mtp","num_speculative_tokens":2}'). MTP runs the model's built-in multi-token-prediction draft heads in parallel with the main forward pass; each accepted speculation multiplies effective throughput. KV-cache compression (--kv-cache-dtype fp8) and the Marlin MoE backend (--moe-backend marlin) round out the configuration.

The new single-request comparison:

The "Strix Halo holds its own on generation" framing from the first benchmark was wrong. With both platforms running the model that actually suits them, with each platform's best decoding pipeline, the H100 is roughly 4.4× faster on raw single-user generation. The earlier 1.4× number was the H100 deliberately handicapped by dense-model arithmetic. Once it gets to use MoE plus MTP, the gap is the gap.

Two specific things drive the uplift. MoE means fewer active parameters per token, so each forward pass is cheaper. MTP means each forward pass can yield two-plus tokens instead of one. Multiply those and you get the 3.5× TPOT improvement at low load (3.6 ms versus 12.8 ms on the dense run), which compounds into the 3.15× single-request output rate versus the same H100's dense numbers.

Concurrency moves further in the same direction

The first sweep showed the H100 scaling to roughly 12× the effective serving capacity of Strix Halo at the 16-agent operating point. With the MoE+MTP configuration, that gap roughly doubles.

The mid-context concurrency shape (4k input, 512 output, agentic-coding turn size):

The sweet spot moves from "8 to 16 agents" to "16 to 32 agents." At concurrency 32 you still get 70 t/s per agent (more than Strix Halo's single-user rate) and 2,229 t/s aggregate. At concurrency 16 the aggregate is 1,654 t/s with P99 TTFT under 1.7 s, comfortably interactive.

Strix Halo doing the same 16-agent workload still takes roughly 60 seconds (llama.cpp does not batch). The H100 with MoE+MTP does it in roughly 2.5 seconds. The effective serving advantage is now somewhere around 24×, not 12×.

Cost recalculated

Same UpCloud business-hours pricing (€1.79/hr) divided by the new throughput numbers:

At the operating point a small dev team would actually use (16 concurrent agents during work hours), the marginal output cost is roughly €0.30 per million tokens. Anthropic's GPT-5.2-class pricing is now $14 per million output tokens, so the gap is roughly 45×. Even accounting for chain-of-thought overhead on a reasoning model (more on that below) and the fact that real traffic gets lower MTP acceptance than --ignore-eos benchmarks, the economics are not close.

What this means in practice

Two specific deployment shapes still look obviously correct, but the second one looks more obviously correct than I wrote four days ago.

One Strix Halo per developer for personal use, with the right model shape. For one human at one keyboard running an MoE model that fits the memory architecture, Strix Halo remains a reasonable personal-AI machine. 128 GB unified memory, real bandwidth, no API bill, no data leaving the box. The caveat is still model selection: dense 27B-class models are not what this machine is good at. Pick the MoE variants that suit the bandwidth profile. The Strix Halo setup guide and the gotchas post cover what it takes to actually get there. What the new H100 numbers do change is your expectations of single-user speed: at 54 t/s on Strix Halo versus 236 t/s on a properly-configured H100 endpoint, the H100 is meaningfully snappier to use for the same task. Strix Halo's win is locality and cost, not throughput.

One H100 (rented) per team for multi-agent backends. The moment your workflow runs more than one agent at a time, the architecture gap is the gap that matters, and it is now even bigger. vLLM's continuous batching plus MoE plus MTP turns one GPU into a serving fleet that absorbs ~2,200 output tokens per second at the operational ceiling. pi-ensemble dispatches up to six specialist children per /work invocation; on Strix Halo those run sequentially, on an H100 they run concurrently, and at 16-agent concurrency four-to-six developers can each run their own pi-ensemble simultaneously without anyone noticing.

This is the shape we are using at Trail Openers. One shared H100 in UpCloud's Helsinki data centre, internal endpoints behind Caddy, business-hours uptime. The decision to host locally rather than reach for a hyperscaler GPU instance was driven by three things at once: the sustainability footprint (UpCloud's energy mix is majority renewable and their ESG reporting covers scope 1, 2, and 3 with compensation for scope 1+2), the data-jurisdiction story (everything stays in the EU, GDPR-native, no extra-territorial transfers), and the simple fact that the machine sits in the same metropolitan area as the team using it. Latency from a developer's desk in greater Helsinki to a model running in fi-hel2 is dominated by the local fibre hop, not by any cross-continent route. None of this makes inference free of footprint (it never is), but it shifts the marginal cost of an extra agent run onto a cleaner grid, in a friendlier jurisdiction, with materially better latency than the default AWS/GCP region you would otherwise reach.

The caveats are real

Five things to know before you act on this.

The 35B-A3B variant trades a bit of reasoning quality for throughput. Qwen's own SWE-bench numbers put the dense 27B at ~77% and the 35B-A3B at ~73%. For complex multi-step code debugging the dense model is still the better tool. For chat, summarisation, retrieval-augmented Q&A, and the bulk of agentic-coding work, the MoE+MTP combination wins on every operational axis. Pick the model by workload; do not assume one is universally better.

MTP acceptance is workload-dependent. The 3.15× single-request uplift in the table above is on --ignore-eos random-token traffic, which is unusually easy speculation. Real chat workloads, especially code generation with strict syntax, see lower acceptance rates and therefore lower uplift. Plan for 2.0× to 2.5× sustained uplift in production rather than the 3.15× peak.

Reasoning models burn most of the output budget on chain-of-thought. Qwen3.6 emits roughly 80% of every response as reasoning trace before the final answer. The 512-token output budgets in the tables above are mostly thinking, not result. If you want 200 tokens of actual answer, plan for 1,000-1,500 output tokens per agent. A non-reasoning model, or chat_template_kwargs.enable_thinking: false, would shift this picture significantly.

P99 TTFT under load still has a tail. At concurrency 16 the median TTFT is 283 ms but the P99 is 1.62 s. At concurrency 32 the P99 is 3.17 s. This is substantially better than the dense run (which hit P99 of 7.45 s at concurrency 32), but the variance still grows with load. If interactive consistency matters more than aggregate throughput, cap at 16.

First-deploy cold start is expensive. torch.compile takes about 20 minutes on first run. Subsequent boots reuse the cache and reach /health 200 in roughly 2 minutes. Worth knowing if you spin instances up and down.

The takeaway

I came into the first benchmark expecting the H100 to be dramatically faster at everything. The first sweep, with the dense 27B, suggested it was only 1.4× faster on single-user generation, and I wrote that up. That conclusion was wrong, and the way it was wrong is instructive. Comparing different-shape models across different hardware does not isolate the hardware. It tells you what your model choice is doing.

When both platforms run the same MoE variant with each platform's best decoding stack, the H100 is roughly 4.4× faster on single-user generation, roughly 8× faster on prefill, and roughly 24× more effective at concurrent serving. The architectural gap I described four days ago is real and bigger than I said. The 1.4× number should not have been the headline.

For our use at Trail Openers, the conclusion is sharper than before. Strix Halo on the desk for personal work, running MoE models that fit the bandwidth profile. A shared H100 in UpCloud's Helsinki data centre during business hours, serving the team's multi-agent backends with the sustainability footprint, EU data residency, and same-city latency we wanted. Same shape of answer as before, with a clearer view of how big the gap actually is when you compare like for like.

Benchmark details: vLLM 0.21.0. Dense 27B run (2026-06-01): max_model_len=8192, max_num_seqs=256, gpu_memory_utilization=0.9. MoE+MTP run (2026-06-05): max_num_seqs=128, gpu_memory_utilization=0.85, kv_cache_dtype=fp8, moe_backend=marlin, reasoning_parser=qwen3, speculative_config={"method":"mtp","num_speculative_tokens":2}. Both swept with vllm bench serve --dataset-name random --ignore-eos. Hardware: UpCloud H100 80GB SXM, single GPU, fi-hel2. Strix Halo numbers from my own internal benchmark notes, best of ROCm pr21344 / Vulkan RADV / ROCm 7.2.3 backends.

I spent half of yesterday watching a pair-coding setup fail at the same task three times in a row. The setup was the one I had been quietly proud of: a developer agent and an adversarial-developer agent running concurrently, with the adversary observing the developer's live stream and able to interrupt mid-task whenever it spotted a problem. The premise felt obvious. Why wait for a bad diff when you can catch the mistake while it is being made?

In theory, an attractive idea. In practice, after many rounds of tweaking the prompts, debouncing rules, and interrupt semantics, the result has been the same every time: a confused developer agent that produces no valuable output. Yesterday was the cleanest example I have. A 16-minute session that burned 13.68M tokens and ended with a working tree full of .bak files, plus two follow-up sessions that produced zero code edits across 38 combined developer turns. The pattern was clean enough that I went back to read what the rest of the field has been doing with coder + critic agent pairs. The answer was unkind: the architecture I had built is exactly the one the production literature has been moving away from for the past year. This is the story of what went wrong, why, and what I should have been doing instead, which is the same thing I have already been doing in production for about a year.

The setup

The system is pi-ensemble, an extension I maintain that wraps the Pi terminal coding agent and turns the parent process into a project manager dispatching role-specialised child processes. One of those tools is pair_watch. It spawns a developer child and an adversarial-developer child simultaneously, summarises each developer turn into ≤500 characters of tool-call descriptions and message excerpts, and pipes that summary into the adversary as a steering input. The adversary can then call interrupt_developer whenever it sees something concerning. That call is injected back into the developer's next turn as a user-message prefixed [pair:adversarial].

The developer's system prompt tells it to "read the interrupt before your next action" and "adjust your plan." The adversary's prompt says, almost verbatim, "Restraint is false economy. If you can predict a failure mode the dev hasn't addressed, interrupt." You can probably guess where this is going.

The task that broke it

The work itself was a six-pass code review verdict on a real PR with three remaining MEDIUM findings. Two were architectural (a chat SSE contract that had started carrying research-specific event types, and a deep_research function that had grown a direct coupling to the streaming transport) and one was error-handling (an unknown-status branch that returned an error without emitting a terminal Failed event to the UI). The first two findings touched signatures in a Rust file using async tokio channels, trait objects, and lifetimes. If you have done that kind of refactor you already know what is coming: it is not difficult conceptually, but it requires holding several constraint sets in your head at once. Lifetimes that satisfy one call site need to also satisfy the other. The compile-error feedback loop is slow and the errors are walls of text.

I dispatched pair_watch. Sixteen minutes later it came back with a verdict of ESCALATED and a working tree in worse shape than before.

What the transcripts showed

I pulled the developer and adversarial JSON transcripts and walked through them turn by turn. Two distinct failure modes appeared, and they are worth separating because they have different causes.

Session 1 was the architecturally pure failure. The developer made real progress on the two simpler findings within the first four minutes. Then it hit the trait-object refactor. The Rust file oscillated in size, 671 → 854 → 622 → 671 lines, as the developer started a structural edit, was interrupted by the adversary mid-edit ("the old function body wasn't fully removed; the file has duplicate function definitions"), abandoned the partial work to address the interrupt, started a different approach, was interrupted again ("the file is now 866 lines, over the 500-line limit"), and so on. By the end the file contained the old function body, a half-finished make_progress_callback replacement, and an uncompilable type alias. The developer re-read the same file 35 times across 112 turns, about 31% of all turns. Input-to-output token ratio was 4.4:1, which is what you would expect when context is being rebuilt after every interruption rather than flowing forward.

What is important about session 1 is that every adversarial interrupt was correct. The duplicate code really was there. The file size really was over budget. The signature mismatch really did exist. The interrupts were good signal arriving at the wrong moment. Each one forced the developer to leave a half-finished structural edit and respond, which is exactly how you end up with three half-finished structural edits laid on top of each other.

Sessions 2 and 3 were a different failure, and I want to be careful to distinguish them. After the first session left a messy partial state, I asked the system to recover. This time the developer made zero edits across 38 turns: all colgrep, ctx7 documentation lookups, file-line counts, and re-reads. The adversary interrupted nine times across the two sessions, each interrupt becoming progressively more pointed ("Please stop searching and implement"). The developer did not implement.

This second mode is tempting to blame on the interrupts but I do not think that is quite right. The developer was already in an avoidance loop before the interrupts started. The working tree was incoherent, two files were mid-refactor in incompatible ways, and the model in use (a smaller one I had switched to mid-recovery) did not want to commit to an edit it might have to undo. The interrupts were trying to push the developer out of the loop, not into it. The second-order point is the one that matters: session 1's interrupt-driven thrash created the broken state that sessions 2 and 3 could not recover from. The live-steering mechanism produced an artifact, a half-edited working tree, that subsequent runs inherited as input.

Why the architecture is biased this way

I went back and read my own code, which is always a humbling exercise. The mechanism is doing exactly what it was designed to do. The design is the problem.

Three things compound. First, each interrupt is injected as a user-turn in the developer's context. There is no debouncing, no minimum gap. If the adversary sees a problem after every developer turn, the developer gets an interrupt after every turn. Second, the developer's prompt explicitly instructs it to re-plan on interrupt. So the developer treats every interrupt as a signal to pivot, not as a note to file. Third, the adversary's prompt is biased toward firing: "restraint is false economy." Combine these and you have a system that, by construction, prevents the developer from sustaining a multi-turn structural edit.

For tasks that decompose into independent steps (a small bug fix, an incremental refactor, a feature with a clear scaffolding) this is fine and probably helpful. The pivot cost is small and the catch is valuable. For tasks where the steps do not decompose, anything where you have to hold N constraints simultaneously and resolve them with a single coherent edit, every pivot is a partial-write that has to be unwound or merged. The pivot cost dominates the catch value.

What the field already knew

After enough self-flagellation I went looking for who else had tried this. The literature is more developed than I expected and the convergence is striking.

The dominant pattern in production multi-agent work is generate-then-critique with a debate loop. The MASQRAD paper from early 2025 is representative (the domain is data visualization queries rather than code, but the mechanism transfers): an actor LLM produces the full artifact, then a critic LLM enters a multi-agent debate to refine it. The critic does not interrupt generation. This is, almost exactly, the legacy developer → adversarial_loop flow that pair_watch was meant to replace.

A more recent paper, MASDP in IEEE TSE (Jan 2026), is the one that hits closest to home. The authors explicitly identify the two failure modes that emerge when critic-style agents try to steer coder agents: "the difficulty in accurately interpreting complex role prompts" and "the fragility of inter-agent coordination." They propose a different architecture entirely. Rather than a reactive critic, they fine-tune an anticipatory "Reminder" agent that frontloads likely-failure warnings into the coder's initial prompt and iteratively refines those precautions based on execution feedback. The result outperforms GPT-4 baselines while reducing compute, on small LLMs. The phrasing in the paper is striking: they describe their contribution as shifting optimisation burden off the programmer agent. That is the inverse of what live steering does, which is to add burden. Every interrupt is a new context to integrate.

A2C-LLM (MDPI Drones, May 2026) goes further in a different domain (UAV swarm task allocation): replace the LLM critic with a lightweight value-function head. The critic produces a scalar advantage signal, not English-language commentary. This is cheaper, more stable, and does not pollute the actor's context window. The principle generalises beyond drones, but it is only feasible when you have something to optimise toward, which for general code work you usually do not.

On the framework level, the divergence is now explicit. AutoGen, the conversational turn-by-turn steering framework, is widely characterised as a research and brainstorming tool with a steeper learning curve and higher debugging difficulty that makes it ill-suited for production. LangGraph has become the production default specifically because it is deterministic, checkpoints state, and exposes observability. The community has been making this call out loud for at least a year, and I missed it because the pair-coding metaphor was too seductive.

The unifying lesson across all of these is the same: anticipatory critique beats reactive interruption. Tell the coder what to watch for before it starts. Review the result after it finishes. Do not talk to it while it is thinking.

What I am taking from this

Three honest things.

The first is the empirical one. For a year I have been running the turn-based pattern: the developer agent works in one context, the adversarial reviewer runs after the diff is produced in a separate context, and the verdict feeds either a fix loop or a merge. That setup works. It has been the backbone of every serious piece of agentic work I have shipped. Pair-watch was an attempt to improve on it by moving the critique earlier, and across many rounds of tuning it has not produced a single result that the turn-based pattern would not have produced more cheaply and more reliably. The architecture is the problem, not the prompts.

The second is the meta-point, which is the one I keep returning to. The reason pair_watch keeps being seductive is that "two agents working together in real time" sounds intuitively better than "one agent works and then another reviews." It pattern-matches to how humans pair-program well. But LLMs are not humans. They do not have the working memory to hold a multi-constraint problem across an interruption and pick it up where they left off. They re-plan from scratch every turn, and re-planning from scratch with one more user-turn worth of context is worse, not better, than re-planning from scratch with the original problem statement. The thing that makes human pair programming work, shared continuous state, is the thing LLMs structurally do not have. No amount of prompt tweaking changes that.

The third is what I am actually going to do. Pair-watch is gone from the default workflow. The default returns to where it has been for a year: generate, then adversarial-review, then six-pass lens review. The mechanism stays in the repo because there are tasks where the pivot cost is low and an early "stop, you're about to do X" is cheap insurance: linear bug fixes, walking through a series of small independent edits, anything where the plan does not need to bend. For those, opt-in. For everything else, the live channel between the two agents is a tax on the developer's attention.

The next experiment is the MASDP pattern from the IEEE TSE paper: front-load the adversary's predictable warnings into the developer's initial prompt as anticipatory constraints and let the developer work through them without interruption. Whether that improves on plain post-hoc review I genuinely do not know yet. At least now I know what hypothesis I am testing, and which one has already failed.

If you maintain a multi-agent system that does live steering and you have seen it converge on hard problems, I would like to hear about it. Particularly the wall-clock-per-task numbers and what kinds of tasks it handles well. The literature I found was largely against the pattern. The counterexamples might be more interesting than the consensus.

I released pi-ensemble yesterday. It is alpha. The pattern it codifies is not.

For more than a year I have been running a multi-agent setup against my forked opencode: a project manager orchestrating specialist children, a mandatory adversarial gate before commits, a six-lens code review before merges. The combination is slower than letting a single agent rip through tickets, and it is not subtle about it. That is the whole point. pi-ensemble is the same workflow rebuilt as a clean extension to Pi, Mario Zechner's terminal coding agent. Same philosophy, fewer hacks.

The thing most agentic coding setups get wrong

The default optimization target for an AI coding agent right now is velocity. Lines of code per hour. Tickets closed per day. Time from prompt to PR. The frameworks lean into it: parallel workers, autonomous loops, "ship it" rhetoric. The metric is throughput.

This is the wrong metric if you care about whether the code is correct.

Codebases age. The cost of a bug found in review is a fraction of the cost of the same bug found in production. The cost of a security issue found by an adversarial pass is a fraction of the cost of the same issue found by Snyk in your dependency tree. The cost of a poorly-typed signature surfaced before merge is a fraction of the cost of refactoring around it three months later. Speed at the input does not save time at the output. It just shifts where you pay.

pi-ensemble is built on the opposite assumption: that the right number of agents in a workflow is "however many it takes to find what is wrong before you commit it."

The architecture

The parent pi you launch is the project manager. When you run a slash command, the extension injects PM doctrine into the system prompt for that turn. The PM then dispatches specialist children: each one is a separate Pi process spawned with pi --mode json -p --no-extensions --no-session --append-system-prompt <role.md>, with its own assembled prompt and its own context window.

Six roles ship:

Children do not share context with the PM. They report back through structured tool calls. This keeps each specialist's context small and focused, and prevents the slow context contamination that single-agent workflows accumulate over a long session.

Five commands

/start          Initialise session: memory, codebase index, git/PR/CI state
/research       Fan out explore specialists in parallel
/plan           Draft a GitHub issue, classify, apply template
/work           Run an issue end-to-end
/review         On-demand six-pass review of a PR or path

The interesting one is /work. Hand it an issue number and it runs the full pipeline: feature branch, optional parallel worktrees, developer dispatches to implement, mandatory adversarial gate, ops commits, PR, six-pass code review, CI watch, merge per AGENTS.md policy. The two gates are not optional. They are the reason the setup exists.

Two gates

The adversarial gate runs before every commit. An adversarial-developer child receives the diff and the implementation context and is given one job: find what is wrong. Edge cases, missing error handling, off-by-one errors, security implications, behavioural assumptions that do not hold. If it finds something, the developer gets up to three rounds of fixes. Only then does the commit happen.

This catches a class of bugs that single-agent setups miss systematically. A single agent that wrote the code is the wrong agent to evaluate it: it carries the same assumptions, the same blind spots, the same confidence about what should work. An adversarial child with a different system prompt and no context contamination finds things the writer cannot see.

The six-pass code review runs before merge. Six children, each pinned to one lens:

Findings come back as schema-validated report_finding tool calls. They get deduplicated by (path, line, title), precedence-merged so the highest severity for a given location wins, and turned into a verdict: APPROVED, ISSUES_FOUND, or CRITICAL_ISSUES_FOUND. The merge does not happen on a critical verdict without explicit override.

Six lenses are not arbitrary. They are the categories that I have seen single-agent reviews most consistently miss, distilled from a year of opencode runs and several thousand findings logged in the corresponding skill files.

What is underneath

pi-ensemble does not stand alone. It assumes a stack:

• vipune for cross-session memory. Every agent calls it.
• oo for context-efficient wrapping of chatty CLIs like git and gh. Without this the specialist windows fill with noise.
• colgrep for semantic code search. Used to find existing implementations before writing new ones.
• parallel-cli for web search and deep research. The explore role expects it.

This is the visible top of a longer-running effort. Each of these tools exists because something in the workflow needed it and the existing options were not good enough.

Per-role models

You almost certainly want a smarter model for the PM and a faster one for the specialists. pi-ensemble has a 5-layer resolution for subagent models, from per-call override down to a global default. Run /ensemble-model to pick interactively from whichever providers you have authenticated through Pi's /login (Anthropic, OpenAI, GitHub Copilot, Cerebras, whatever).

A typical configuration: Opus or Sonnet on the PM, fast Cerebras models on the lens reviewers, Sonnet on the developer and adversarial-developer. The cost arithmetic is roughly six children × two-thousand-token outputs plus context, per review cycle, which lands around $0.02 to $0.10 on Cerebras and considerably more on Anthropic. That is the tax for the quality gate.

Why Pi, not opencode

I have been running this same orchestrator pattern against my opencode fork for over a year. The maintenance cost of that fork is what eventually pushed me to migrate.

Opencode is a great piece of software. It is also a much larger surface area than I needed: a TUI, a web UI, a desktop wrapper, a server, a plugin layer, multiple packages. Every upstream change required reconciling against my own modifications, every dependency bump touched something I had to retest. The deeper I got into customising the orchestration layer, the more time I spent on integration rather than on the workflow itself.

Pi sits in a different place. The harness is lightweight: a single binary, a small system prompt that ships from the harness rather than being layered into your config, an extension model that is straightforward to develop against. There are no async tasks in either harness yet (this is the one thing I miss most), but Pi feels faster and gets out of the way more readily.

The ecosystem dynamics are also different. Anomaly has a real business model around opencode involving their Zen LLM API and hosted services, and the contribution rules reflect that. The current CONTRIBUTING.md is more open than it used to be (the old policy was effectively "no feature PRs from outside the core team"), but UI and core product changes still require a design review with the core team before implementation. For a fork that needs to evolve at the pace of my own experimentation, that is the wrong governance model. Not wrong in absolute terms; wrong for what I am trying to do.

Pi's extension story lets me ship the orchestration layer cleanly without forking the harness at all. pi-ensemble is an extension. So is pi-worktree. So is pi-permissions. Each is a separate concern, each can move on its own schedule, none of them require me to maintain a fork of the underlying tool. That is the part I could not get with opencode without spending most of my evenings on rebase.

If opencode is the right tool for you, keep using it. The work I did there is what made pi-ensemble possible. But the maintenance arithmetic stopped working for me, and the migration has paid for itself already.

The honest caveats

This is alpha. Things will change before 1.0. Specifically:

Permissions are not enforced per role. Specialists inherit Pi's default permissions. The role system prompt is the only thing keeping each in its lane. This is acceptable on a sandbox repo and not acceptable on anything you care about. pi-permissions will fix this.

The six-pass review costs real money. Not enough to matter on personal projects. Enough to matter if you fire it indiscriminately across a hundred PRs a day. Pin the cheap models for the lens reviewers; reserve the expensive ones for the PM.

Worktrees go through git CLI calls. They work, but the safer path is the pi-worktree plugin when its programmatic API stabilises.

It is tested on macOS. Linux should work. Windows almost certainly does not.

This is not for everyone. If your team measures developer output in story points and your AI integration is supposed to make those story points cheaper, pi-ensemble is the wrong tool. It will spend more tokens, take longer, and produce fewer PRs per day. What it produces will be better-reviewed, more defensible, and less likely to bite you later. That is a trade you have to actively want.

Why this exists

A year of running this pattern in opencode taught me that the bottleneck in agentic coding is not the model. It is the discipline applied to what the model produces. Models will happily ship reasonable-looking code that is wrong in non-obvious ways. The question is whether your workflow gives that code a serious chance of being caught.

pi-ensemble is one answer to that question. The interfaces will change. The philosophy will not.

Repo: github.com/randomm/pi-ensemble