Can the biggest, smartest model that physically fits on a Strix Halo box earn a place in the daily rotation? DeepSeek-V4-Flash at IQ2_M is 91 GB, right at the edge of what 128 GB of unified memory allows. It benchmarks like a frontier model: MMLU-Pro 86, GPQA 88, SWE-bench 79. If it ran at usable speed, it would be the best local model this hardware can hold.

So I benchmarked it. It runs. Here is how fast.

Update 2026-08-07: DeepSeek has since shipped V4-Flash-0731, a re-post-trained official release of the model benchmarked here. The speed numbers below should carry over; the quality picture changes. See the update at the end.

The setup

Everything below is measured on my Strix Halo box (AMD Ryzen AI MAX+ 395, 128 GB unified memory, the machine from the setup guide). Model: DeepSeek-V4-Flash, IQ2_M quant, 91 GB on disk. Flags: -fa 1, as always on this machine.

One hard constraint up front: the KV cache must be f16. Quantizing it to q8_0 produces garbage output on this architecture. That is not a tuning preference, it is mandatory, and it costs you memory headroom you do not have much of at 91 GB of weights.

ROCm vs Vulkan

Both backends load and run the model cleanly. No crashes, flash attention works. But they are not equal:

Vulkan wins generation by 33%. For a chat or agentic model, generation speed is the number that matters, so Vulkan is the backend for this model. ROCm only looked competitive on a tiny 64-token prefill; at realistic prompt sizes Vulkan pulls ahead there too.

Generation holds steady at roughly 13 t/s across context. That is below the ~19 t/s figure cited in the research around this model. My guess: that number came from a different quant or from the custom ROCmFPX pipeline, not from stock llama.cpp on gfx1151.

The prefill problem

Generation is slow but usable. Prefill is the real problem.

Prefill drops from 148 t/s at 512 tokens of context to 103 t/s at 4096. It keeps degrading from there. A full-matrix run appeared to hang on the pp16384 test. It had not hung. It was genuinely grinding along at an extrapolated ~50 t/s.

Do the arithmetic on an agentic workload. A 16K-token prompt at ~50 t/s prefill is minutes of waiting before the first output token. Every tool call that re-submits context pays that price again. My suspicion is that DeepSeek-V4's sparse-attention and indexer kernels simply are not well optimized on gfx1151 yet, so the architecture's efficiency tricks turn into overhead here.

Where it lands in the lineup

This makes DeepSeek-V4-Flash the slowest big model on the box, not the crown jewel:

Qwen3.5-122B is twice as fast, half the size on disk, and leaves real memory headroom instead of running at the ceiling. At 91 GB plus f16 KV cache, DeepSeek-V4-Flash leaves almost nothing spare, and running that close to the limit is exactly where this machine's memory-pressure failure modes live.

Honest verdict

Feasible but not practical. That is the whole finding.

DeepSeek-V4-Flash is the highest-quality model that fits on this hardware, and you pay for that quality with 13 t/s generation, prefill that collapses on long prompts, and a memory footprint that crowds out everything else. As a daily workhorse it loses to Qwen3.5-122B on every operational axis. The "big quality" default does not change.

Where it could earn a slot: a rare, load-on-demand "I need the single best answer and I will wait" mode. Short-context reasoning tasks, where its weak prefill does not bite and its benchmark-topping quality does. Wired into llama-swap as an occasional route with Vulkan and f16 KV, evicting everything else first. Not for agentic work. Not for long context. Not for anything interactive.

There is a general lesson in here for local inference on this class of hardware. Fitting is not the bar. A model can load, run cleanly, and pass every smoke test, and still be the wrong choice because the tokens-per-second economics do not work for how you actually use it. Measure generation speed, measure prefill at the prompt sizes your real workloads produce, then decide. The best model you can fit is not automatically the best model you can use.

Update 2026-08-07: V4-Flash-0731 changes the quality math

Five days after this post, DeepSeek released DeepSeek-V4-Flash-0731, the official release that supersedes the preview benchmarked above. Same architecture, same 284B total / 13B active size. All the gains come from re-post-training, which means the speed numbers in this post should transfer unchanged: same weights footprint, same prefill behavior, same ~13 t/s.

What changed is quality, and by a lot if the vendor numbers hold. Per DeepSeek's launch table, 0731 beats the larger V4-Pro Preview on every agentic and coding benchmark they published: Terminal Bench 82.7 vs 72.1, DeepSWE 54.4 vs 12.8, NL2Repo 54.2 vs 38.5. It lands near Opus-class agentic territory while remaining behind Opus 4.8 on every row. The usual caveats apply: vendor-reported, unreleased harness, and BenchLM notes the widely-quoted Terminal-Bench jump compares two different benchmark versions.

For this box, the verdict shifts in one direction only. The practicality problems are architectural, so they stay: prefill still collapses on long prompts, the memory ceiling is still the memory ceiling, and agentic use is still ruled out. But the "load-on-demand, I need the single best answer and I will wait" niche just got meaningfully stronger, because the quality you are waiting for is now higher, especially for coding. Two things to watch before re-testing: Unsloth shipped Q4 and Q8 GGUFs on day one, but sub-100 GB quants in this post's IQ2_M territory were still pending as of early August, and the DSpark speculative-decode module that ships with the model is not yet supported in llama.cpp. When a small quant lands, the numbers above are the baseline to beat.

Laguna-S-2.1 is the agentic-coding model of the moment. poolside released it on 2026-07-21: 118B total parameters, ~8B active MoE, "most capable in its weight class." The hype is all H100s and DGX Sparks.

I run it on an AMD Ryzen AI MAX+ 395 mini-PC. Radeon 8060S integrated GPU, gfx1151, 128 GB of unified LPDDR5X. This is the "can the cheap unified-memory box really run it?" story, with receipts.

Day one, and it just works

llama.cpp merged the laguna architecture two days after release (PR #25165, 2026-07-22). A stock Vulkan build loaded the model within hours of me pulling it.

The serving stack is the same one from the setup guide: llama-swap hot-swapping per-model llama.cpp instances, pre-built gfx1151 toolboxes from kyuz0/amd-strix-halo-toolboxes. Download the 73 GB UD-Q4_K_XL GGUF, drop a conf, add a route, restart. Live.

The quiet miracle is worth stating plainly: a 73 GB model with 256K context across 3 slots, all resident in unified memory on an integrated GPU. No consumer discrete GPU can hold this. You would need multiple cards. That is the whole Strix Halo thesis in one screenshot.

The numbers

Measured on my box, UD-Q4_K_XL, Vulkan RADV, llama.cpp b10118, production flags (-fa 1 --no-mmap -ctk q8_0 -ctv q8_0):

~74 GB resident. Decode barely degrades with context: 30 t/s cold, 27.4 t/s at 16K in. Prefill holds up too, only dropping from 399 to 343 t/s across the same range.

For context, the head-to-head against Qwen3.6-35B-A3B on the same box:

Teaching moment #1: it is all in the active parameters. The ~3× prefill gap and ~1.5× decode gap are not mysterious. 8B active versus 3B active is a 2.7× ratio, and that lands almost exactly on the measured prefill difference. Decode is cushioned by memory bandwidth. Once you internalize "speed is a function of active parameters, not total parameters," every MoE number on this hardware becomes predictable.

The gotchas

Three hard-won bits that make this post worth bookmarking.

1. Vulkan only. The ROCm backend crashes. On gfx1151, HIP flash-attention has no device code for Laguna's head-dim-128 sliding-window layers:

fattn-mma-f16.cuh: no device code compatible with HIP arch 1300

Vulkan RADV handles it fine. Another Strix Halo user confirmed the same crash in the PR thread, so it is not my build.

2. Thinking is load-bearing. Laguna interleaves reasoning with output, and quality craters if you truncate it. The GGUF ships a max_new_tokens default that can cut thinking short. Raise it, and keep enable_thinking on for anything hard.

3. Loading needs a clean GPU. A 73 GB weight load plus staging leaves no room for leftovers. An orphaned model from a prior run caused an ErrorDeviceLost on my first attempt. More on where that road leads below.

The rabbit hole: can we make it faster?

Most posts stop at "it runs." I spent the session trying to make it faster and failed three times. The failures are more instructive than the successes.

ngram speculative decoding: a beautiful net-negative. On a verbatim-echo prompt it hit 94.9 t/s, a 3.2× speedup. On a real code-edit turn it regressed decode to 20.5 t/s, a 32% loss, and on novel reasoning it was neutral. Teaching moment #2: speculative decoding only pays when draft acceptance is high. Real agentic edits diverge from the existing context enough that the draft overhead becomes dead weight. Rejected.

Dropping to Q3: the bandwidth trap. Naive math said 26% smaller weights should mean roughly 25% faster decode. Measured gain: 8.6%. Teaching moment #3: Laguna's decode is not purely bandwidth-bound at 8B active. A fixed per-token cost dominates: attention over a huge KV cache, expert routing, and the always-on shared expert. The Q3 dequant kernel eats some of the savings back on top. Not worth the quality risk. Rejected.

DFlash, poolside's own speculative drafter: the tantalizing one. This is the lever that should work, and it is blocked upstream: the draft GGUF will not load on mainline llama.cpp, and there is an open issue (#25117) measuring it roughly 2× slower on a Strix Halo APU. Community numbers, not mine. So the one real upside lever is both unavailable and possibly counterproductive on this hardware. Watching, not waiting.

Verdict: ~30 t/s is near this silicon's ceiling for an 8B-active MoE with dense attention over a large KV cache. That is a physics-grounded conclusion, not a tuning failure.

The war story

Mid-experiments, I ran a big model load concurrently with a big download. Free RAM hit ~2 GB, and model loads started wedging inside the GPU sub-allocator instead of completing. Then each llama-swap retry piled another one on. The signature:

• 8 processes stuck in uninterruptible D-state, wchan = drm_suballoc_new. Unkillable by any signal. It is a deadlock: memory cannot free because the holders are themselves waiting for memory.
• GTT pinned at 67 GB while llama-swap reported nothing loaded.
• No GPU hang in dmesg. Not a crash, an allocator deadlock. Only a reboot cleared it.

The lesson: on unified-memory boxes, serialize your big I/O. Download fully, then load. Never let two large allocations race. The failure mode is not a clean OOM, it is an unkillable deadlock.

The takeaways

• Yes, a ~€4K integrated-GPU mini-PC runs a frontier-class 118B agentic coder at a genuinely usable ~30 t/s, holding 256K context in unified memory. Two years ago this needed a multi-GPU rig.
• Routing wisdom: use Laguna for long-horizon, terminal-driven agentic work where one better decision saves round-trips. Use a lighter MoE like Qwen3.6-35B for interactive, high-frequency loops. Different tools.
• The unified-memory superpower is capacity, not speed. You will not out-decode a 5090, but you will fit things a 5090 cannot, and for local agentic coding, fitting the model plus huge context beats raw t/s.
• Be honest about the ceiling: prefill ~3× slower than a small MoE, no working speculative decoding, Vulkan-only. All fixable upstream over time.

Reproducibility

Every number above is first-party, measured on my box this week. The DFlash slowdown and poolside's marketing multiples are community and vendor figures, labeled as such.

• Hardware: Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151, RDNA 3.5, 40 CU), 128 GB LPDDR5X-8000 (~220 GB/s real).
• Stack: kyuz0/amd-strix-halo-toolboxes (vulkan-radv), llama.cpp b10118, llama-swap.
• Model: unsloth/Laguna-S-2.1-GGUF UD-Q4_K_XL (73 GB, 3 parts).
• Flags: -ngl 999 -fa 1 --no-mmap -ctk q8_0 -ctv q8_0 --kv-unified --parallel 3 -c 262144 --jinja. Vulkan, not ROCm.
• Bench: llama-batched-bench with -npp 512,4096,16384 -ntg 128 -npl 1.

The setup itself is covered in the Strix Halo setup guide and the gotchas post.

For the last two weeks my feed has been "loop engineering" this, "loop engineering" that. Addy Osmani named it on June 7. Within ten days there were follow-ups from Cobus Greyling, Lushbinary, MindStudio, Louis-François Bouchard, Kilo, Firecrawl, several YouTube videos, an Instagram reel, and a Reddit thread asking whether it is just the next buzzword. Two industry figures got cited everywhere. Boris Cherny, who leads Claude Code at Anthropic: "I don't prompt Claude anymore. I have loops running. My job is to write loops." Peter Steinberger, creator of OpenClaw: "You shouldn't be prompting coding agents anymore. You should be designing loops that prompt your agents."

I read enough of these to figure out what was being claimed, and then I had a slightly disorienting realisation. By every definition in those articles, I have been loop engineering for over a year. At one point I actually asked Claude whether we should throw some loop engineering into pi-ensemble. Claude pointed out, with the patience of someone explaining something obvious, that pi-ensemble already is loop engineering.

What the term actually means

The framing that has settled out across the articles is a three-floor stack. Prompt engineering is the ground floor: write a good prompt for a single turn. Harness engineering is the middle floor: design the environment a single agent runs inside (its tools, its context, its rubric). Loop engineering is the top floor: design the system that prompts the harness for you. It runs on a schedule. It spawns sub-agents. It verifies its own output. It decides whether to keep going. The model becomes a subroutine inside your loop, not a chat partner on the other side of a prompt box.

The four-step cycle inside the loop is the same in every article: act, observe, reason, repeat. The articles also converge on the same structural ingredient as the thing that actually makes loops work, which is splitting the maker from the checker. The model that wrote the code is too charitable about its own output. A second agent with a different system prompt, and ideally a different model, catches what the first one talked itself into. Sub-agents in .claude/agents/ (Claude Code) and .codex/agents/ (OpenAI Codex) are the productised primitive for this. Addy Osmani makes it the centrepiece of his post. Boris Cherny describes it as how he actually works. The articles call this the heart of the practice.

The pattern is real. The term is also real, and it does shift the conversation usefully: the leverage point moves from writing prompts to designing the system that writes them.

What I was already running

This is the part that was disorienting.

For the last year my main coding environment has been a forked opencode with a custom multi-agent configuration. The shape: one parent process acting as a project manager, dispatching to specialist children. Developer. Adversarial reviewer. Ops. Explore. Code-review children, one per lens (security, error handling, type safety, performance, architecture, simplicity). The PM holds the workflow state. The children do the work and report back. Nothing in the loop talks to me on a per-turn basis. I give it an issue or a directive; it runs through plan, work, gate, review, until it has produced something to merge, or it has hit something it cannot handle and has to escalate.

This year I rebuilt the whole thing as a clean Pi extension called pi-ensemble. Same architecture, less fork maintenance. The five slash commands cover the cycle Osmani describes almost line for line:

• /start initialises the session: searches memory, indexes the codebase, gathers git/PR/CI state. Discovery.
• /research fans out explore specialists in parallel. Context.
• /plan drafts and classifies a GitHub issue. Intent.
• /work runs the full pipeline: branch, developer, mandatory adversarial gate (up to 3 fix rounds), commit, PR, six-pass code review, CI watch, merge. Act, observe, verify, repeat.
• /review runs the six-lens review on demand against any PR or path.

The maker/checker split that Osmani says is the most useful structural thing in a loop is, in pi-ensemble, two separate gates. The adversarial-developer child gets the diff before any commit and tries to break it. Three rounds of fix-and-retry. If it survives that, the six lens reviewers run in parallel, each pinned to its lens, and the findings get deduplicated and precedence-merged into a verdict. Merge does not happen on a critical verdict without override.

I built none of this because anyone called it loop engineering. I built it because turn-by-turn babysitting of a coding agent on hard tasks does not work, and I needed a system that could grind through real PRs without me holding its hand. The pattern emerged from the problem. I am sitting at three screens, up to 6-7 separate sessions and burning hundreds of millions of tokens a day. It would not be possible if I had to be constantly involved with every decision in every session.

What was actually new about the term

The pattern is older than the term. Geoffrey Huntley's "Ralph" technique (early 2026, before there was a name for any of this) is a one-line shell loop that feeds the same prompt to a fresh agent until a status file says done. The articles correctly cite Ralph as the prior art. My setup is a more structured version of the same idea, with named roles and explicit gates instead of one prompt and a status file. Many other practitioners landed on similar shapes independently. The Anthropic Effective harnesses for long-running agents write-up describes the same primitives. OpenAI's Symphony is a fleet-management layer over the same cycle.

What the term does is consolidate a lot of small individual realisations into one named thing that the field can argue about. That is not nothing. Before the name, you had to spend a paragraph explaining what you were doing. After the name, you can point at the stack and say "this is the loop part" and most people understand. Naming things compresses the discourse, and a compressed discourse moves faster.

The other thing the term does is force the maker/checker question to the front. A lot of the early agentic coding hype was "one big agent that does everything." The loop engineering framing makes it obvious that the interesting design choices are about the structure of the loop, not the capability of the single agent. That is the right place for the leverage to be.

What the articles get wrong

Two things, mostly minor.

First, the articles tend to treat the maker/checker split as something you bolt onto a single-agent setup. In practice, the more useful framing is that the loop is multi-agent by construction. The PM is not an enhanced single agent. It is a different kind of agent, with a different job, that happens to dispatch other agents. Treating the orchestrator as first-class changes the questions you ask about the system.

Second, the cost numbers in the new posts are wild. A six-pass code review at frontier-model rates per PR adds up fast. The H100 in production economics make this more defensible, but the articles tend to gloss the operating envelope. Loop engineering only pays for itself when the loop produces something worth its token budget, which is much harder than getting the loop to run.

What I am taking from this

Mostly that the term is useful enough that I will start using it. "Pi-ensemble is my loop engineering setup" is shorter than what I used to have to say.

The deeper thing is the same observation that comes up every time the field names a pattern that practitioners were already running. The naming compresses the discourse, but it also resets the apparent frontier. Articles dated June 7 onward get framed as "the new wave." Setups that were doing the same thing in March or April look like prior art. There is a slight unfairness in how the credit lands, and a slightly larger unfairness in how the buyer-facing narrative settles ("this just emerged"). Neither is the term's fault. The pattern is older than the name, and the people who needed the pattern figured it out before there was a name for it.

If you are reading the loop engineering articles and thinking "this looks like what I have been doing," you are probably right. The discourse caught up. That is good. Use the name. Cite the framing. And do not be surprised that the actual work, the rubrics inside the loop, the verification step, the taste calibration, did not get easier just because there is a term for the box you put it all in.

Mine are not perfect, by the way. Still tuning. Current state at github.com/randomm/pi-ensemble.

I almost added a seventh role to pi-ensemble this week. The reasoning was plausible enough. When the adversarial gate rejects three rounds in a row with overlapping themes, the loop is signalling that the approach is wrong, not the implementation. A fresh "architect" agent could step back and propose a different frame. The PM (Project Manager) would dispatch it on cap-hit. Clean idea. Easy to specify.

I did the research before writing the prompts. The research said no.

What the data shows

The authoritative source is the MAST paper (Cemri et al., NeurIPS 2025): 1,642 execution traces across 7 popular multi-agent frameworks, 14 distinct failure modes, three categories. The headline finding is that failure rates on state-of-the-art multi-agent systems sit between 41% and 86.7%, and that "performance gains often remain minimal compared to single-agent frameworks or simple baselines like best-of-N sampling."

Two of the 14 failure modes are directly relevant to adding a role:

• Disobey Role Specification: 11.8% of all failures. An agent silently behaves like a different agent. The more roles in the system, the more chances for drift.
• Step Repetition: 13.2% of all failures. The orchestrator loses track of what has already been done. The orchestration prompt grows with each role; the orchestrator's grip on it does not.

Add the broader Inter-Agent Misalignment category (31-32% of failures: conversation reset, task derailment, information withholding, ignoring other agents' input, reasoning-action mismatch) and you have an empirical picture that is not subtle. Coordination is where multi-agent systems actually fail. Not capability. Not model choice. Coordination.

The cost of one more role

The intuition I want to displace is that an additional role costs one role's worth of overhead. It does not. Each new role:

• Expands the orchestrator's decision space on every turn (more dispatch conditions to evaluate, more routing combinations to get right)
• Dilutes instruction density in the orchestrator's prompt (the "lost in the middle" phenomenon kicks in earlier when the prompt is busier)
• Adds a compression event for every handoff (the downstream agent sees the output, not the reasoning behind it)
• Creates a new failure surface (every role drift is a potential bug)

The cost is paid every turn the loop runs. The benefit, in the architect-agent case, would be paid only when the adversarial gate hits a cap with thematic overlap. Low-frequency upside against constant-cost downside. The arithmetic does not work.

What I am doing instead

Doctrine, not a new role. The fix is a few paragraphs in the PM's prompt:

• Watch for the pattern: three adversarial rejections with overlapping themes (not orthogonal local bugs)
• When detected, dispatch the existing @explore specialist with a step-back-framed prompt: "Don't review this diff. Consider whether the whole approach is right. Given the original issue and the recurring finding pattern, is there a fundamentally different way to solve this?"
• Take the result, update the spec, surface to user for approval
• Re-enter from /plan with the revised spec

Zero new roles. Existing roles, different prompts. The fresh-context property I wanted from "architect" is already present in @explore (no awareness of the current diff, no role-bias toward defending it or finding bugs in it). The reframe is the prompt, not the role.

This matches what Augment Code's production Coordinator-Specialist-Verifier pattern does. It also matches the recommendation that comes out of MAST: the structural redesign is "removing agents from the coordination role entirely. Agents execute. A governed state machine coordinates." In pi-ensemble, the PM doctrine is the state machine. Doctrine changes are cheap. Roles are expensive.

This is not the first time the doctrine-not-role move has paid off in pi-ensemble. The developer agent already handles what I call the knee method: an agent learning something mid-work that suggests the spec is wrong, and ploughing on into scope it should not be in. Drew Breunig has written the clearest framing of why this matters. His Spec-Driven Development Triangle treats implementation as a feedback mechanism rather than a one-way pipeline: "the act of writing code improves the spec, and it improves the tests." The doctrine in pi-ensemble's developer prompt is the operational counterpart of that idea. Encounter something unexpected that might change the approach, stop and report to the PM, let the PM decide whether the spec needs updating. The agent is the same role. The behaviour is different because the prompt is.

The general rule

If you are tempted to add a role to a multi-agent system, ask whether the same behaviour can be achieved by a different prompt to an existing role. In my experience, the answer is yes most of the time. The exceptions are rare enough that they should be carefully argued for rather than reached for as the default move.

Less is more, in multi-agent setups as elsewhere. The empirical data agrees, which is the more interesting thing than my taste agreeing.

The previous two posts in this series were benchmarks: first sweep on a dense 27B, then the like-for-like rerun on the same MoE variant Strix Halo runs, with MTP speculative decoding. The benchmarks closed the question of "is it fast enough." This post is about the question that comes next: "what does it actually cost to run, in money, energy, and CO₂."

We have now had the H100 endpoint in real use at Trail Openers for about a week. Several developers using it for coding work, not synthetic load. The energy and footprint numbers are nothing like the "H100 = 700W" reflex would predict, and the marginal cost across real coding traffic lands at a small fraction of what an equivalent volume of frontier-API tokens would have cost. This post walks through both, with the caveats they deserve.

What we are actually running

One H100 80GB SXM in UpCloud's fi-hel2 data centre. UpCloud's published per-hour rate during business hours, lower outside. vLLM 0.21.0 serving Qwen3.6-35B-A3B-FP8 (the MoE variant, 3B active out of 35B total) with MTP speculative decoding, FP8 KV-cache, Marlin MoE backend. Endpoint behind Caddy with HTTPS. Business-hours scheduling: the box comes up in the morning, goes down in the evening, weekends off.

The deployment is OpenTofu, idempotent, one tofu apply from cold. The economic and footprint shape depends on the scheduling. Running 24/7 would cost roughly three times what business-hours-only does, for no additional throughput when nobody is at a keyboard. Scheduled correctly, the monthly cost lands in a tight, predictable range.

Energy: well below the TDP

The reflex when you hear "H100" is "700W card." That number is the datasheet TDP, which assumes a particular workload (dense compute, BF16, GPU saturated). What we are running does not look like that workload.

Measured draw from nvidia-smi integrated over time, across a week of real use:

Three reasons the draw stays low. First, the MoE shape: only ~3B of the 35B parameters activate per token, so the compute per token is a fraction of what a dense 35B would burn. Second, FP8 is roughly 2× more energy-efficient than BF16 for the same arithmetic. Third, vLLM's prefix caching eliminates re-computation across conversational turns, which removes a category of work that would otherwise consume tokens and energy for no marginal benefit.

A live calibration confirmed the meter is unbiased (no methodology bug; the low number is real for this workload). The H100 is not a 700W card in the way most people imagine. It is a 700W card running below 50% utilization for this kind of inference, which is the same as saying it is a ~330W card when it matters.

CO₂: single-digit grams per hour

Helsinki sits on one of the cleanest electricity grids in Europe. Finland's lifecycle factor in May 2026 was 54 gCO₂/kWh per Electricity Maps. Apply that to the measured energy draw with a PUE of 1.2:

The actual week of data confirms the range. Looking at our busiest billed hour (2026-06-15 09:00 UTC, 109M input tokens through the endpoint): 0.317 kWh, 17.1 gCO₂. Most working hours land in the 5-9 gCO₂ range.

Project that to a month of business-hours operation: roughly 2.7-4.6 kg of CO₂. This is comparable to running a household refrigerator for a few weeks, not to anything that should give anyone climate anxiety. The reason is not that AI inference is magically clean. It is that the specific combination of MoE + FP8 + Helsinki grid + business-hours scheduling sits at the favourable end of every variable that determines the footprint.

The UpCloud fi-hel2 facility additionally runs on 100% renewable energy and feeds waste heat into the district heating network (the operator I have been able to identify serves up to ~28,000 homes from this and adjacent facilities). The marginal kilowatt of compute, on top of being clean at the input side, displaces heating fuel at the output side. None of which makes inference free of footprint. It just shifts where the offset comes from.

An important caveat. The CO₂ numbers are estimated, not live-measured. We are using a constant grid factor (54 gCO₂/kWh) and a constant PUE (1.2). Both vary in reality. The energy figures from nvidia-smi are exact (the GPU's total_energy_consumption counter, sampled to a database every five minutes). The carbon translation on top is reasonable but not certified.

Cost: a fixed ceiling instead of a meter

The interesting property of the cost picture is not the absolute number. It is the shape. A rented dedicated GPU costs what it costs whether the team writes one diff or a hundred. There is no surprise bill, no per-token meter spinning faster as the workload scales. For a team that does not yet know how heavily it will use its agents in any given week, that is a structurally different financial risk profile than paying per token to a frontier API.

The marginal cost across real coding traffic comes out well below current frontier-API rates, and well below the published rates for hosted open-weight inference of the same model. The exact ratios depend on the comparison and the load shape, both covered in the next section. The takeaway for budgeting is simpler: instead of an unbounded line item that scales with usage, you get a predictable monthly figure that lands in roughly the same range regardless of how heavily the box gets driven within the working day.

Versus the alternatives

This is where the picture sharpens. Two comparisons that matter:

Versus Anthropic Sonnet 4.6. At our current load shape, our marginal cost is roughly 15-18× cheaper per million tokens than Sonnet's published rates. But the headline ratio understates the difference for the actual shape of agentic coding traffic, which is dramatically input-heavy. The ratio of input to output tokens in our real usage is around 120:1. The agent reads a lot of code and writes a small diff. On real two-hour samples of our actual workload, the same traffic priced on Sonnet would have cost roughly 23-74× more than running it on our own H100, depending on whether the hour was light or heavy. Frontier APIs bleed on input tokens, and agentic coding is the workload where that bleed hurts most.

Versus a hosted open-weight API serving the same Qwen3.6-35B-A3B model. Hosted open-weight inference of this model is priced an order of magnitude below Sonnet, so the gap narrows. In the near-idle state we are roughly at parity. In busy hours, where our utilisation rises and our marginal output cost drops, we are roughly 3.5× cheaper than the hosted alternative. The price advantage of the self-hosted option grows with utilisation. Below a certain steady-state load the hosted API is the right answer; above it, the rented dedicated GPU wins.

This is the part of the picture that surprises people: open-weight models on hosted APIs have already collapsed most of the price gap to running them yourself. The dominant remaining argument for self-hosting is not "it is much cheaper." It is the structural properties: data sovereignty, fixed cost ceiling, predictable monthly accounting, and the ability to integrate the inference endpoint into the same network and trust boundary as the rest of the infrastructure.

The honest caveats

Five things to know before you read these numbers as a guarantee.

This is early operational data. A week of real but light-to-moderate use with some test traffic mixed in. Not a sustained steady-state under heavy 16-agent multi-team load. The benchmarks suggest the operating economics get better at higher utilisation (marginal cost per output token drops), but I cannot show you a month of that yet.

The CO₂ numbers are estimated, not live-measured. Constant 54 gCO₂/kWh Finland factor, constant 1.2 PUE. Both vary in reality; both are reasonable approximations.

MTP acceptance in production is lower than benchmark. The 3.15× single-request uplift in the benchmarks was on --ignore-eos random-token traffic. Real chat workloads see 2.0-2.5× sustained. Already factored into the operational numbers above, just worth saying out loud.

Business-hours scheduling has real ergonomic costs. You cannot run a long-running agent task overnight if the box is down. We have specific workflows that need this (memory consolidation, batch reviews) and we either schedule them to fit the window or accept a 24/7 cost premium for the specific hours we need.

The Trail Openers context is specific. EU jurisdiction, GDPR concerns, the team's physical location matching the data centre, the company's sustainability stance: these are real reasons for us that may or may not be reasons for you. The economic argument generalises better than the locality argument.

What this changes

For Trail Openers, this confirms the architecture decision. The shared H100 in Helsinki is meaningfully cheaper than the alternatives we were comparing against. The monthly cost ceiling is predictable. The footprint is small and on a grid that is cleaner than nearly any hyperscaler default region. And because everything stays in fi-hel2 and on internal endpoints, the data-sovereignty story is clean.

For anyone evaluating a similar setup: the headline economics are real but the durable arguments are structural. A predictable monthly bill instead of an open per-token meter. EU data residency by construction, not by configuration. Clean grid at the input, heat recovery at the output, single-digit kg of CO₂ per month at our scale. The interesting question is not whether self-hosting is cheap. It is whether the structural properties are worth the operational work, and at what team size the answer flips.

For a team of four-to-six developers doing agentic coding, our experience so far is that the answer flipped some time ago.

Telemetry source: nvidia-smi total_energy_consumption (exact GPU counter, driver 595.58.03), sampled to a database every five minutes, then aggregated into hourly usage reports. Cost figures from UpCloud's published per-hour rates. Energy-to-CO₂ translation: constant 54 gCO₂/kWh (Finland lifecycle, Electricity Maps May 2026) × constant 1.2 PUE.

I have been benchmarking Qwen3.6 on an UpCloud H100 80GB SXM, comparing it against the numbers I have been running on my Strix Halo box. The first sweep, four days ago, used Qwen3.6-27B-FP8 (dense) on the H100 under vLLM 0.21, and my established Qwen3.6-35B MoE Q8 numbers on Strix Halo under llama.cpp. The directional answer it gave me was "single-user generation is closer than you would think; prefill and concurrency win." That answer turns out to have been half-right and half-misleading. I ran the like-for-like comparison today and want to correct the picture.

The context for this work has not changed. At Trail Openers we are moving toward more environmentally sustainable LLM infrastructure for our internal tooling, and the H100 setup we are building will be shared by several developers. UpCloud was chosen for three reasons that compound. First, their data centres run majority on renewable energy and their scope 1+2 emissions are compensated. Second, the fi-hel2 data centre is in Helsinki, the same metropolitan area as Trail Openers, which means the whole team gets single-digit-millisecond latency to the endpoint. Third, EU jurisdiction and GDPR-native data handling matter for the work we do; the prompts, the diffs, the codebases, none of that leaves the EU.

What the first sweep got wrong

The first benchmark ran a dense 27B on the H100 against an MoE on Strix Halo. That comparison was fair on weight footprint (28-34 GB class) but unfair on architectural fit. Dense and MoE behave very differently per-token, and the model choice on each side biased the result in opposite directions.

The summary numbers from that first sweep:

The 1.4× looked like good news for Strix Halo. It is not. It is the artefact of two model choices, not a real architectural finding. Strix Halo was running the MoE variant that suits its ~225 GB/s bandwidth (only the active experts have to be streamed per token). The H100 was running a dense model where every parameter is touched on every forward pass. That is not a fair fight in either direction.

The fair comparison is running the same MoE variant on both, with each platform's best serving stack and best decoding tricks. So I did that.

The like-for-like sweep

Yesterday I redeployed the H100 with Qwen3.6-35B-A3B-FP8 (the same 3B-active / 35B-total MoE Strix Halo runs) plus MTP speculative decoding turned on (--speculative-config '{"method":"mtp","num_speculative_tokens":2}'). MTP runs the model's built-in multi-token-prediction draft heads in parallel with the main forward pass; each accepted speculation multiplies effective throughput. KV-cache compression (--kv-cache-dtype fp8) and the Marlin MoE backend (--moe-backend marlin) round out the configuration.

The new single-request comparison:

The "Strix Halo holds its own on generation" framing from the first benchmark was wrong. With both platforms running the model that actually suits them, with each platform's best decoding pipeline, the H100 is roughly 4.4× faster on raw single-user generation. The earlier 1.4× number was the H100 deliberately handicapped by dense-model arithmetic. Once it gets to use MoE plus MTP, the gap is the gap.

Two specific things drive the uplift. MoE means fewer active parameters per token, so each forward pass is cheaper. MTP means each forward pass can yield two-plus tokens instead of one. Multiply those and you get the 3.5× TPOT improvement at low load (3.6 ms versus 12.8 ms on the dense run), which compounds into the 3.15× single-request output rate versus the same H100's dense numbers.

Concurrency moves further in the same direction

The first sweep showed the H100 scaling to roughly 12× the effective serving capacity of Strix Halo at the 16-agent operating point. With the MoE+MTP configuration, that gap roughly doubles.

The mid-context concurrency shape (4k input, 512 output, agentic-coding turn size):

The sweet spot moves from "8 to 16 agents" to "16 to 32 agents." At concurrency 32 you still get 70 t/s per agent (more than Strix Halo's single-user rate) and 2,229 t/s aggregate. At concurrency 16 the aggregate is 1,654 t/s with P99 TTFT under 1.7 s, comfortably interactive.

Strix Halo doing the same 16-agent workload still takes roughly 60 seconds (llama.cpp does not batch). The H100 with MoE+MTP does it in roughly 2.5 seconds. The effective serving advantage is now somewhere around 24×, not 12×.

Cost recalculated

Same UpCloud business-hours pricing (€1.79/hr) divided by the new throughput numbers:

At the operating point a small dev team would actually use (16 concurrent agents during work hours), the marginal output cost is roughly €0.30 per million tokens. Anthropic's GPT-5.2-class pricing is now $14 per million output tokens, so the gap is roughly 45×. Even accounting for chain-of-thought overhead on a reasoning model (more on that below) and the fact that real traffic gets lower MTP acceptance than --ignore-eos benchmarks, the economics are not close.

What this means in practice

Two specific deployment shapes still look obviously correct, but the second one looks more obviously correct than I wrote four days ago.

One Strix Halo per developer for personal use, with the right model shape. For one human at one keyboard running an MoE model that fits the memory architecture, Strix Halo remains a reasonable personal-AI machine. 128 GB unified memory, real bandwidth, no API bill, no data leaving the box. The caveat is still model selection: dense 27B-class models are not what this machine is good at. Pick the MoE variants that suit the bandwidth profile. The Strix Halo setup guide and the gotchas post cover what it takes to actually get there. What the new H100 numbers do change is your expectations of single-user speed: at 54 t/s on Strix Halo versus 236 t/s on a properly-configured H100 endpoint, the H100 is meaningfully snappier to use for the same task. Strix Halo's win is locality and cost, not throughput.

One H100 (rented) per team for multi-agent backends. The moment your workflow runs more than one agent at a time, the architecture gap is the gap that matters, and it is now even bigger. vLLM's continuous batching plus MoE plus MTP turns one GPU into a serving fleet that absorbs ~2,200 output tokens per second at the operational ceiling. pi-ensemble dispatches up to six specialist children per /work invocation; on Strix Halo those run sequentially, on an H100 they run concurrently, and at 16-agent concurrency four-to-six developers can each run their own pi-ensemble simultaneously without anyone noticing.

This is the shape we are using at Trail Openers. One shared H100 in UpCloud's Helsinki data centre, internal endpoints behind Caddy, business-hours uptime. The decision to host locally rather than reach for a hyperscaler GPU instance was driven by three things at once: the sustainability footprint (UpCloud's energy mix is majority renewable and their ESG reporting covers scope 1, 2, and 3 with compensation for scope 1+2), the data-jurisdiction story (everything stays in the EU, GDPR-native, no extra-territorial transfers), and the simple fact that the machine sits in the same metropolitan area as the team using it. Latency from a developer's desk in greater Helsinki to a model running in fi-hel2 is dominated by the local fibre hop, not by any cross-continent route. None of this makes inference free of footprint (it never is), but it shifts the marginal cost of an extra agent run onto a cleaner grid, in a friendlier jurisdiction, with materially better latency than the default AWS/GCP region you would otherwise reach.

The caveats are real

Five things to know before you act on this.

The 35B-A3B variant trades a bit of reasoning quality for throughput. Qwen's own SWE-bench numbers put the dense 27B at ~77% and the 35B-A3B at ~73%. For complex multi-step code debugging the dense model is still the better tool. For chat, summarisation, retrieval-augmented Q&A, and the bulk of agentic-coding work, the MoE+MTP combination wins on every operational axis. Pick the model by workload; do not assume one is universally better.

MTP acceptance is workload-dependent. The 3.15× single-request uplift in the table above is on --ignore-eos random-token traffic, which is unusually easy speculation. Real chat workloads, especially code generation with strict syntax, see lower acceptance rates and therefore lower uplift. Plan for 2.0× to 2.5× sustained uplift in production rather than the 3.15× peak.

Reasoning models burn most of the output budget on chain-of-thought. Qwen3.6 emits roughly 80% of every response as reasoning trace before the final answer. The 512-token output budgets in the tables above are mostly thinking, not result. If you want 200 tokens of actual answer, plan for 1,000-1,500 output tokens per agent. A non-reasoning model, or chat_template_kwargs.enable_thinking: false, would shift this picture significantly.

P99 TTFT under load still has a tail. At concurrency 16 the median TTFT is 283 ms but the P99 is 1.62 s. At concurrency 32 the P99 is 3.17 s. This is substantially better than the dense run (which hit P99 of 7.45 s at concurrency 32), but the variance still grows with load. If interactive consistency matters more than aggregate throughput, cap at 16.

First-deploy cold start is expensive. torch.compile takes about 20 minutes on first run. Subsequent boots reuse the cache and reach /health 200 in roughly 2 minutes. Worth knowing if you spin instances up and down.

The takeaway

I came into the first benchmark expecting the H100 to be dramatically faster at everything. The first sweep, with the dense 27B, suggested it was only 1.4× faster on single-user generation, and I wrote that up. That conclusion was wrong, and the way it was wrong is instructive. Comparing different-shape models across different hardware does not isolate the hardware. It tells you what your model choice is doing.

When both platforms run the same MoE variant with each platform's best decoding stack, the H100 is roughly 4.4× faster on single-user generation, roughly 8× faster on prefill, and roughly 24× more effective at concurrent serving. The architectural gap I described four days ago is real and bigger than I said. The 1.4× number should not have been the headline.

For our use at Trail Openers, the conclusion is sharper than before. Strix Halo on the desk for personal work, running MoE models that fit the bandwidth profile. A shared H100 in UpCloud's Helsinki data centre during business hours, serving the team's multi-agent backends with the sustainability footprint, EU data residency, and same-city latency we wanted. Same shape of answer as before, with a clearer view of how big the gap actually is when you compare like for like.

Benchmark details: vLLM 0.21.0. Dense 27B run (2026-06-01): max_model_len=8192, max_num_seqs=256, gpu_memory_utilization=0.9. MoE+MTP run (2026-06-05): max_num_seqs=128, gpu_memory_utilization=0.85, kv_cache_dtype=fp8, moe_backend=marlin, reasoning_parser=qwen3, speculative_config={"method":"mtp","num_speculative_tokens":2}. Both swept with vllm bench serve --dataset-name random --ignore-eos. Hardware: UpCloud H100 80GB SXM, single GPU, fi-hel2. Strix Halo numbers from my own internal benchmark notes, best of ROCm pr21344 / Vulkan RADV / ROCm 7.2.3 backends.

I spent half of yesterday watching a pair-coding setup fail at the same task three times in a row. The setup was the one I had been quietly proud of: a developer agent and an adversarial-developer agent running concurrently, with the adversary observing the developer's live stream and able to interrupt mid-task whenever it spotted a problem. The premise felt obvious. Why wait for a bad diff when you can catch the mistake while it is being made?

In theory, an attractive idea. In practice, after many rounds of tweaking the prompts, debouncing rules, and interrupt semantics, the result has been the same every time: a confused developer agent that produces no valuable output. Yesterday was the cleanest example I have. A 16-minute session that burned 13.68M tokens and ended with a working tree full of .bak files, plus two follow-up sessions that produced zero code edits across 38 combined developer turns. The pattern was clean enough that I went back to read what the rest of the field has been doing with coder + critic agent pairs. The answer was unkind: the architecture I had built is exactly the one the production literature has been moving away from for the past year. This is the story of what went wrong, why, and what I should have been doing instead, which is the same thing I have already been doing in production for about a year.

The setup

The system is pi-ensemble, an extension I maintain that wraps the Pi terminal coding agent and turns the parent process into a project manager dispatching role-specialised child processes. One of those tools is pair_watch. It spawns a developer child and an adversarial-developer child simultaneously, summarises each developer turn into ≤500 characters of tool-call descriptions and message excerpts, and pipes that summary into the adversary as a steering input. The adversary can then call interrupt_developer whenever it sees something concerning. That call is injected back into the developer's next turn as a user-message prefixed [pair:adversarial].

The developer's system prompt tells it to "read the interrupt before your next action" and "adjust your plan." The adversary's prompt says, almost verbatim, "Restraint is false economy. If you can predict a failure mode the dev hasn't addressed, interrupt." You can probably guess where this is going.

The task that broke it

The work itself was a six-pass code review verdict on a real PR with three remaining MEDIUM findings. Two were architectural (a chat SSE contract that had started carrying research-specific event types, and a deep_research function that had grown a direct coupling to the streaming transport) and one was error-handling (an unknown-status branch that returned an error without emitting a terminal Failed event to the UI). The first two findings touched signatures in a Rust file using async tokio channels, trait objects, and lifetimes. If you have done that kind of refactor you already know what is coming: it is not difficult conceptually, but it requires holding several constraint sets in your head at once. Lifetimes that satisfy one call site need to also satisfy the other. The compile-error feedback loop is slow and the errors are walls of text.

I dispatched pair_watch. Sixteen minutes later it came back with a verdict of ESCALATED and a working tree in worse shape than before.

What the transcripts showed

I pulled the developer and adversarial JSON transcripts and walked through them turn by turn. Two distinct failure modes appeared, and they are worth separating because they have different causes.

Session 1 was the architecturally pure failure. The developer made real progress on the two simpler findings within the first four minutes. Then it hit the trait-object refactor. The Rust file oscillated in size, 671 → 854 → 622 → 671 lines, as the developer started a structural edit, was interrupted by the adversary mid-edit ("the old function body wasn't fully removed; the file has duplicate function definitions"), abandoned the partial work to address the interrupt, started a different approach, was interrupted again ("the file is now 866 lines, over the 500-line limit"), and so on. By the end the file contained the old function body, a half-finished make_progress_callback replacement, and an uncompilable type alias. The developer re-read the same file 35 times across 112 turns, about 31% of all turns. Input-to-output token ratio was 4.4:1, which is what you would expect when context is being rebuilt after every interruption rather than flowing forward.

What is important about session 1 is that every adversarial interrupt was correct. The duplicate code really was there. The file size really was over budget. The signature mismatch really did exist. The interrupts were good signal arriving at the wrong moment. Each one forced the developer to leave a half-finished structural edit and respond, which is exactly how you end up with three half-finished structural edits laid on top of each other.

Sessions 2 and 3 were a different failure, and I want to be careful to distinguish them. After the first session left a messy partial state, I asked the system to recover. This time the developer made zero edits across 38 turns: all colgrep, ctx7 documentation lookups, file-line counts, and re-reads. The adversary interrupted nine times across the two sessions, each interrupt becoming progressively more pointed ("Please stop searching and implement"). The developer did not implement.

This second mode is tempting to blame on the interrupts but I do not think that is quite right. The developer was already in an avoidance loop before the interrupts started. The working tree was incoherent, two files were mid-refactor in incompatible ways, and the model in use (a smaller one I had switched to mid-recovery) did not want to commit to an edit it might have to undo. The interrupts were trying to push the developer out of the loop, not into it. The second-order point is the one that matters: session 1's interrupt-driven thrash created the broken state that sessions 2 and 3 could not recover from. The live-steering mechanism produced an artifact, a half-edited working tree, that subsequent runs inherited as input.

Why the architecture is biased this way

I went back and read my own code, which is always a humbling exercise. The mechanism is doing exactly what it was designed to do. The design is the problem.

Three things compound. First, each interrupt is injected as a user-turn in the developer's context. There is no debouncing, no minimum gap. If the adversary sees a problem after every developer turn, the developer gets an interrupt after every turn. Second, the developer's prompt explicitly instructs it to re-plan on interrupt. So the developer treats every interrupt as a signal to pivot, not as a note to file. Third, the adversary's prompt is biased toward firing: "restraint is false economy." Combine these and you have a system that, by construction, prevents the developer from sustaining a multi-turn structural edit.

For tasks that decompose into independent steps (a small bug fix, an incremental refactor, a feature with a clear scaffolding) this is fine and probably helpful. The pivot cost is small and the catch is valuable. For tasks where the steps do not decompose, anything where you have to hold N constraints simultaneously and resolve them with a single coherent edit, every pivot is a partial-write that has to be unwound or merged. The pivot cost dominates the catch value.

What the field already knew

After enough self-flagellation I went looking for who else had tried this. The literature is more developed than I expected and the convergence is striking.

The dominant pattern in production multi-agent work is generate-then-critique with a debate loop. The MASQRAD paper from early 2025 is representative (the domain is data visualization queries rather than code, but the mechanism transfers): an actor LLM produces the full artifact, then a critic LLM enters a multi-agent debate to refine it. The critic does not interrupt generation. This is, almost exactly, the legacy developer → adversarial_loop flow that pair_watch was meant to replace.

A more recent paper, MASDP in IEEE TSE (Jan 2026), is the one that hits closest to home. The authors explicitly identify the two failure modes that emerge when critic-style agents try to steer coder agents: "the difficulty in accurately interpreting complex role prompts" and "the fragility of inter-agent coordination." They propose a different architecture entirely. Rather than a reactive critic, they fine-tune an anticipatory "Reminder" agent that frontloads likely-failure warnings into the coder's initial prompt and iteratively refines those precautions based on execution feedback. The result outperforms GPT-4 baselines while reducing compute, on small LLMs. The phrasing in the paper is striking: they describe their contribution as shifting optimisation burden off the programmer agent. That is the inverse of what live steering does, which is to add burden. Every interrupt is a new context to integrate.

A2C-LLM (MDPI Drones, May 2026) goes further in a different domain (UAV swarm task allocation): replace the LLM critic with a lightweight value-function head. The critic produces a scalar advantage signal, not English-language commentary. This is cheaper, more stable, and does not pollute the actor's context window. The principle generalises beyond drones, but it is only feasible when you have something to optimise toward, which for general code work you usually do not.

On the framework level, the divergence is now explicit. AutoGen, the conversational turn-by-turn steering framework, is widely characterised as a research and brainstorming tool with a steeper learning curve and higher debugging difficulty that makes it ill-suited for production. LangGraph has become the production default specifically because it is deterministic, checkpoints state, and exposes observability. The community has been making this call out loud for at least a year, and I missed it because the pair-coding metaphor was too seductive.

The unifying lesson across all of these is the same: anticipatory critique beats reactive interruption. Tell the coder what to watch for before it starts. Review the result after it finishes. Do not talk to it while it is thinking.

What I am taking from this

Three honest things.

The first is the empirical one. For a year I have been running the turn-based pattern: the developer agent works in one context, the adversarial reviewer runs after the diff is produced in a separate context, and the verdict feeds either a fix loop or a merge. That setup works. It has been the backbone of every serious piece of agentic work I have shipped. Pair-watch was an attempt to improve on it by moving the critique earlier, and across many rounds of tuning it has not produced a single result that the turn-based pattern would not have produced more cheaply and more reliably. The architecture is the problem, not the prompts.

The second is the meta-point, which is the one I keep returning to. The reason pair_watch keeps being seductive is that "two agents working together in real time" sounds intuitively better than "one agent works and then another reviews." It pattern-matches to how humans pair-program well. But LLMs are not humans. They do not have the working memory to hold a multi-constraint problem across an interruption and pick it up where they left off. They re-plan from scratch every turn, and re-planning from scratch with one more user-turn worth of context is worse, not better, than re-planning from scratch with the original problem statement. The thing that makes human pair programming work, shared continuous state, is the thing LLMs structurally do not have. No amount of prompt tweaking changes that.

The third is what I am actually going to do. Pair-watch is gone from the default workflow. The default returns to where it has been for a year: generate, then adversarial-review, then six-pass lens review. The mechanism stays in the repo because there are tasks where the pivot cost is low and an early "stop, you're about to do X" is cheap insurance: linear bug fixes, walking through a series of small independent edits, anything where the plan does not need to bend. For those, opt-in. For everything else, the live channel between the two agents is a tax on the developer's attention.

The next experiment is the MASDP pattern from the IEEE TSE paper: front-load the adversary's predictable warnings into the developer's initial prompt as anticipatory constraints and let the developer work through them without interruption. Whether that improves on plain post-hoc review I genuinely do not know yet. At least now I know what hypothesis I am testing, and which one has already failed.

If you maintain a multi-agent system that does live steering and you have seen it converge on hard problems, I would like to hear about it. Particularly the wall-clock-per-task numbers and what kinds of tasks it handles well. The literature I found was largely against the pattern. The counterexamples might be more interesting than the consensus.

I released pi-ensemble yesterday. It is alpha. The pattern it codifies is not.

For more than a year I have been running a multi-agent setup against my forked opencode: a project manager orchestrating specialist children, a mandatory adversarial gate before commits, a six-lens code review before merges. The combination is slower than letting a single agent rip through tickets, and it is not subtle about it. That is the whole point. pi-ensemble is the same workflow rebuilt as a clean extension to Pi, Mario Zechner's terminal coding agent. Same philosophy, fewer hacks.

The thing most agentic coding setups get wrong

The default optimization target for an AI coding agent right now is velocity. Lines of code per hour. Tickets closed per day. Time from prompt to PR. The frameworks lean into it: parallel workers, autonomous loops, "ship it" rhetoric. The metric is throughput.

This is the wrong metric if you care about whether the code is correct.

Codebases age. The cost of a bug found in review is a fraction of the cost of the same bug found in production. The cost of a security issue found by an adversarial pass is a fraction of the cost of the same issue found by Snyk in your dependency tree. The cost of a poorly-typed signature surfaced before merge is a fraction of the cost of refactoring around it three months later. Speed at the input does not save time at the output. It just shifts where you pay.

pi-ensemble is built on the opposite assumption: that the right number of agents in a workflow is "however many it takes to find what is wrong before you commit it."

The architecture

The parent pi you launch is the project manager. When you run a slash command, the extension injects PM doctrine into the system prompt for that turn. The PM then dispatches specialist children: each one is a separate Pi process spawned with pi --mode json -p --no-extensions --no-session --append-system-prompt <role.md>, with its own assembled prompt and its own context window.

Six roles ship:

Children do not share context with the PM. They report back through structured tool calls. This keeps each specialist's context small and focused, and prevents the slow context contamination that single-agent workflows accumulate over a long session.

Five commands

/start          Initialise session: memory, codebase index, git/PR/CI state
/research       Fan out explore specialists in parallel
/plan           Draft a GitHub issue, classify, apply template
/work           Run an issue end-to-end
/review         On-demand six-pass review of a PR or path

The interesting one is /work. Hand it an issue number and it runs the full pipeline: feature branch, optional parallel worktrees, developer dispatches to implement, mandatory adversarial gate, ops commits, PR, six-pass code review, CI watch, merge per AGENTS.md policy. The two gates are not optional. They are the reason the setup exists.

Two gates

The adversarial gate runs before every commit. An adversarial-developer child receives the diff and the implementation context and is given one job: find what is wrong. Edge cases, missing error handling, off-by-one errors, security implications, behavioural assumptions that do not hold. If it finds something, the developer gets up to three rounds of fixes. Only then does the commit happen.

This catches a class of bugs that single-agent setups miss systematically. A single agent that wrote the code is the wrong agent to evaluate it: it carries the same assumptions, the same blind spots, the same confidence about what should work. An adversarial child with a different system prompt and no context contamination finds things the writer cannot see.

The six-pass code review runs before merge. Six children, each pinned to one lens:

Findings come back as schema-validated report_finding tool calls. They get deduplicated by (path, line, title), precedence-merged so the highest severity for a given location wins, and turned into a verdict: APPROVED, ISSUES_FOUND, or CRITICAL_ISSUES_FOUND. The merge does not happen on a critical verdict without explicit override.

Six lenses are not arbitrary. They are the categories that I have seen single-agent reviews most consistently miss, distilled from a year of opencode runs and several thousand findings logged in the corresponding skill files.

What is underneath

pi-ensemble does not stand alone. It assumes a stack:

• vipune for cross-session memory. Every agent calls it.
• oo for context-efficient wrapping of chatty CLIs like git and gh. Without this the specialist windows fill with noise.
• colgrep for semantic code search. Used to find existing implementations before writing new ones.
• parallel-cli for web search and deep research. The explore role expects it.

This is the visible top of a longer-running effort. Each of these tools exists because something in the workflow needed it and the existing options were not good enough.

Per-role models

You almost certainly want a smarter model for the PM and a faster one for the specialists. pi-ensemble has a 5-layer resolution for subagent models, from per-call override down to a global default. Run /ensemble-model to pick interactively from whichever providers you have authenticated through Pi's /login (Anthropic, OpenAI, GitHub Copilot, Cerebras, whatever).

A typical configuration: Opus or Sonnet on the PM, fast Cerebras models on the lens reviewers, Sonnet on the developer and adversarial-developer. The cost arithmetic is roughly six children × two-thousand-token outputs plus context, per review cycle, which lands around $0.02 to $0.10 on Cerebras and considerably more on Anthropic. That is the tax for the quality gate.

Why Pi, not opencode

I have been running this same orchestrator pattern against my opencode fork for over a year. The maintenance cost of that fork is what eventually pushed me to migrate.

Opencode is a great piece of software. It is also a much larger surface area than I needed: a TUI, a web UI, a desktop wrapper, a server, a plugin layer, multiple packages. Every upstream change required reconciling against my own modifications, every dependency bump touched something I had to retest. The deeper I got into customising the orchestration layer, the more time I spent on integration rather than on the workflow itself.

Pi sits in a different place. The harness is lightweight: a single binary, a small system prompt that ships from the harness rather than being layered into your config, an extension model that is straightforward to develop against. There are no async tasks in either harness yet (this is the one thing I miss most), but Pi feels faster and gets out of the way more readily.

The ecosystem dynamics are also different. Anomaly has a real business model around opencode involving their Zen LLM API and hosted services, and the contribution rules reflect that. The current CONTRIBUTING.md is more open than it used to be (the old policy was effectively "no feature PRs from outside the core team"), but UI and core product changes still require a design review with the core team before implementation. For a fork that needs to evolve at the pace of my own experimentation, that is the wrong governance model. Not wrong in absolute terms; wrong for what I am trying to do.

Pi's extension story lets me ship the orchestration layer cleanly without forking the harness at all. pi-ensemble is an extension. So is pi-worktree. So is pi-permissions. Each is a separate concern, each can move on its own schedule, none of them require me to maintain a fork of the underlying tool. That is the part I could not get with opencode without spending most of my evenings on rebase.

If opencode is the right tool for you, keep using it. The work I did there is what made pi-ensemble possible. But the maintenance arithmetic stopped working for me, and the migration has paid for itself already.

The honest caveats

This is alpha. Things will change before 1.0. Specifically:

Permissions are not enforced per role. Specialists inherit Pi's default permissions. The role system prompt is the only thing keeping each in its lane. This is acceptable on a sandbox repo and not acceptable on anything you care about. pi-permissions will fix this.

The six-pass review costs real money. Not enough to matter on personal projects. Enough to matter if you fire it indiscriminately across a hundred PRs a day. Pin the cheap models for the lens reviewers; reserve the expensive ones for the PM.

Worktrees go through git CLI calls. They work, but the safer path is the pi-worktree plugin when its programmatic API stabilises.

It is tested on macOS. Linux should work. Windows almost certainly does not.

This is not for everyone. If your team measures developer output in story points and your AI integration is supposed to make those story points cheaper, pi-ensemble is the wrong tool. It will spend more tokens, take longer, and produce fewer PRs per day. What it produces will be better-reviewed, more defensible, and less likely to bite you later. That is a trade you have to actively want.

Why this exists

A year of running this pattern in opencode taught me that the bottleneck in agentic coding is not the model. It is the discipline applied to what the model produces. Models will happily ship reasonable-looking code that is wrong in non-obvious ways. The question is whether your workflow gives that code a serious chance of being caught.

pi-ensemble is one answer to that question. The interfaces will change. The philosophy will not.

Repo: github.com/randomm/pi-ensemble

This post is overdue. I shipped /add-parallel into NanoClaw in early February. The merge happened the same day I opened the PR. I have been thinking about that experience ever since, not because the PR itself was complicated, but because the contribution model was not normal. Three months on, the pattern has a name (SkDD), a popular framework on top of it (Superpowers, around 95K stars), and a security audit calling it out as a supply chain risk (Snyk's ToxicSkills, 36% of audited skills flagged). It is time to write down what I actually think.

The PR

NanoClaw is Gavriel Cohen's container-isolated alternative to OpenClaw. Around 500 lines of TypeScript on trunk, agents run in actual Linux containers, security is enforced by the OS rather than by application-level allowlists. The contributing rule is exactly five words: don't add features, add skills.

So when I wanted to wire up Parallel AI as an integration, I did not add code to trunk. I wrote a SKILL.md that teaches Claude Code how to transform a NanoClaw fork to include Parallel. A user runs /add-parallel in their own checkout, Claude reads the skill, modifies the local code, and the integration is in place. Trunk never sees the diff. My PR added one skill file and a registration entry. That was it.

The strange part was not the code. The strange part was the conceptual flip. The PR did not extend NanoClaw. It added a recipe that lets each user extend their own copy of NanoClaw differently. That is the whole skills-driven development idea in one example.

The methodology, briefly

The pattern was formalized by Zak El Fassi in March under the name SkDD. Every build loop adds one decision gate: should this become a skill? If yes, you write a SKILL.md. The agent finds it, loads it, runs it next time. Three skill types: operational (do discrete work), meta (create other skills), composed (chain skills into pipelines). The compounding happens over months.

Jesse Vincent's Superpowers is the most-adopted implementation for individual development workflows. It bakes TDD, brainstorming, code review, and subagent-driven implementation into a skills library that works across Claude Code, Codex, Cursor, OpenCode, and Gemini CLI. The format converged organically rather than being formally standardized.

NanoClaw is the most opinionated project-level application of the philosophy I have seen. Other projects use skills as a developer's personal productivity layer. NanoClaw uses skills as a contribution model. The codebase deliberately stays minimal because the extensibility lives in the skills branch.

What it gets right

The trunk stays auditable. This is the security argument, and it is real. NanoClaw is small enough to read in an evening. You add the channel, agent provider, or integration you need; you do not inherit the security surface of fifty modules other users wanted. Compare to OpenClaw at ~400K lines: nobody is reviewing that codebase end to end.

The compounding is real for personal repos. I write a skill once. Future-Claude finds it and uses it. The skill survives session boundaries, model swaps, and harness switches. Three months in, I have skills that I no longer consciously remember writing but that get invoked automatically. That is genuine compounding.

It is harness-agnostic. A SKILL.md is just markdown. Claude Code, Codex, Cursor, OpenCode all read it. You are not locking yourself into a vendor's plugin system. This is the most underrated property of the format.

It forces small, composable units. A skill that tries to do too much is hard to write and unreliable to invoke. The format pushes you toward single-responsibility units, which is the same discipline that makes good Unix tools.

Where it breaks

This is the part I do not see written down enough.

Heterogeneous state is the default outcome. When every user runs different /add-* commands against their own fork, no two installations are the same. For a personal AI assistant, that is the point. For a team product, it is a disaster. You cannot debug a deployment when "the codebase" is a hypothesis rather than a fact. You cannot do incident response when the production environment is a snapshot of one developer's skill choices six months ago.

Skills-driven development does not generalize to team production code. It works for personal projects, dev tooling, and individual workflows. It does not work for software multiple humans need to reason about together. The whole point of trunk-based development, code review, and shared conventions is to keep the team's mental model of the system in sync. Skills-driven development inverts that: each fork drifts intentionally.

The output depends on the LLM. Two users running the same /add-telegram skill with Claude Opus 4.7 and GPT-5.2 get different code. Sometimes meaningfully different. The skill is a prompt, not an executable. The result is what the model decides to do with that prompt in that context. For a deterministic build, this is unacceptable. For a personal assistant that gets close enough to what you wanted, it is fine. Know which one you are building.

Skill quality is mostly invisible to the user. Reading a SKILL.md does not tell you what the agent will actually do. The instructions look reasonable. The output may not be. You find out by running it, which is fine on a personal fork and dangerous when the skill touches credentials, deploys code, or modifies shared infrastructure.

The security context is worse than the methodology suggests

Snyk's ToxicSkills audit (February 2026) scanned 3,984 skills from ClawHub and skills.sh. 36.82% had at least one security issue. 13.4% had critical issues including malware, credential theft, and prompt injection. There is no package-signing standard. There is no central review. The format converged faster than the supply chain hygiene around it did.

The lesson is not that skills are bad. The lesson is that "just install this skill" should carry the same suspicion as "just run this curl pipe to bash". Read the skill before you run it. If you would not paste the contents of the SKILL.md into your terminal manually, do not let an agent do it for you.

Where this actually fits

Skills-driven development is a good fit for personal AI assistants (NanoClaw is the proof), individual developer workflows (Superpowers is the proof), and exploratory tooling where each user's needs diverge by design. The compounding is real and the trunk-minimalism is genuinely useful.

It is not a fit for team production code, regulated environments, or anywhere multiple humans need a shared understanding of what the system does. Determinism matters in those contexts, and skills-driven development trades determinism for compounding personal capability. That is a fine trade for one person. It is a bad trade for an organization.

The reason this is worth being explicit about is that the writeups currently in circulation treat SkDD as a general-purpose development methodology. It is not. It is a specific tool for a specific class of problem. For that class, it works well. For everything else, it produces faster chaos.

/add-parallel was the right way to contribute Parallel AI support to NanoClaw. It would have been the wrong way to add Stripe support to a company's billing service. The methodology is not universal. Knowing which side of that line you are on is the part the hype is glossing over.

SkDD methodology | Superpowers | NanoClaw | Snyk ToxicSkills audit

There is no shortage of agent memory systems. mem0, Letta, Zep, Voltropy's LCM, Claude Code's built-in memory, Cursor's context engine. My colleague Topi's Remind is a recent addition to the space. The common pattern across most of these is LLM-based distillation: raw data goes in, an LLM extracts higher-value memories, generalizations, or concepts. mem0 has been doing this for a while. Remind pushes it further with spreading activation retrieval, entity graphs, and outcome tracking, but the core technique is the same.

vipune does not use an LLM. That is a deliberate choice. You store text, it gets embedded locally (ONNX, bge-small-en-v1.5), and you search by meaning. No distillation step, no external model calls, no token cost for memory operations. The tradeoff is obvious: you do not get automatic generalization. What you get is a single binary with no dependencies that runs the same everywhere, offline, with zero configuration.

The 0.5 release adds CLI flags that make this simpler tool useful for things the larger systems were not designed for: scoped multi-agent memory within a single session, typed retrieval, and recency-weighted search that lets you use the same store as both long-term knowledge and short-term working memory.

(I have been toying with the idea of adding an optional distillation step via apfel, the CLI that exposes Apple Intelligence's on-device model on macOS 26+. A local 3B model with no API keys could handle memory consolidation without breaking vipune's zero-dependency, zero-cost model. I have been experimenting with apfel for other things and the on-device inference is fast enough to be practical. But that is future work, not a promise.)

Before getting into the new flags, it is worth looking at one of the more interesting architectural approaches in this space.

What Volt LCM gets right

Voltropy's Volt introduced Lossless Context Management earlier this year. The LCM paper is worth reading. The core insight: stop asking the model to manage its own memory and let the engine do it deterministically. LCM maintains a DAG of hierarchical summaries in a persistent store. Compaction happens asynchronously between turns. Nothing is lost. Sessions can run indefinitely.

The approach is sound. Volt performs well on long-horizon tasks because the model does not have to invent a memory strategy on the fly.

The limitation is structural. Volt is a complete terminal-based coding agent, forked from OpenCode. You get LCM, but you also get the entire agent runtime. If you are building your own harness, or running Claude Code, or using Cursor, Volt's memory is not something you can pull out and use separately.

vipune: just the memory

vipune is the memory layer without the agent. Single binary. No API keys, no daemon, no database server. Everything runs locally using ONNX embeddings (bge-small-en-v1.5). Install it, start using it.

cargo install vipune

Or grab the binary directly:

curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/randomm/vipune/releases/download/v0.5.0/vipune-installer.sh | sh

With 0.5, the feature set covers the main things needed for multi-agent memory workflows. Three capabilities in particular.

Multi-agent scoping

Multiple agents can share a single vipune instance. Inside a git repo, vipune infers the project scope from the repository, so agents working in the same repo share memory by default:

# Both agents are in the same repo — memories are shared automatically
vipune add "Auth service uses JWT with RSA-256"
vipune search "authentication flow"

When you need isolation between agents in the same session, --project overrides the default:

# Agent A: backend scope
vipune --project "myapp/backend" add "Auth service uses JWT with RSA-256"

# Agent B: frontend scope
vipune --project "myapp/frontend" add "Token refresh handled in useAuth hook"

Outside a git repo, set VIPUNE_PROJECT to scope manually. Either way, each agent gets its own memory namespace, or shares one deliberately.

Typed memories

Not all memories are the same. A design decision is not the same thing as a guardrail. vipune now has five memory types:

vipune add "Never deploy to prod on Fridays" --memory-type guard
vipune add "Run migrations before schema tests" --memory-type procedure
vipune search "deployment rules" --memory-type guard,procedure

Agents can filter by type on search. A coding agent looking for guardrails does not need to wade through every factual observation from the last three days. This keeps search results relevant as the memory store grows.

The --status flag adds another axis. Memories start as active or candidate. The --supersedes flag atomically replaces an old memory with a new one:

vipune add "Alice now works at Google" --supersedes abc123-old-memory-id

One transaction. The old memory is marked superseded, the new one is active. No window where both are live.

Tunable recency

Search results combine semantic similarity with a time decay score:

score = (1 - recency_weight) * similarity + recency_weight * time_score

The default is 70% semantic, 30% recency. For long-term project memory, that balance works. For a single programming session where you want recent context to dominate:

vipune search "what did I just decide about the API" --recency 0.8

Now 80% of the ranking comes from how recent the memory is. This is what turns vipune from a knowledge base into working memory. Same binary, same data, different search behaviour depending on what you need right now.

For full-text matching, --hybrid enables BM25 alongside semantic search:

VIPUNE_HYBRID=true vipune search "JWT RSA-256"

Useful when you need exact keyword hits, not just meaning.

MCP server

vipune runs as an MCP server out of the box:

vipune mcp

This exposes store_memory, search_memories, list_memories, and supersede_memory as native tools. Claude Code, Cursor, and anything else that speaks MCP can use vipune as a memory provider without shell command wrappers. The MCP tools accept the same type, status, and filter parameters as the CLI.

Where this fits

The bigger memory systems do more than vipune. Letta has structured agents with memory tiers. Volt LCM has hierarchical DAG compaction. mem0 has managed cloud infrastructure. If you need those things, use those tools.

The niche vipune occupies is narrower. You have an agent (or several), you want them to remember things across turns or sessions, and you do not want to add a service, a daemon, or an account to make that happen. You want a binary you can call from a shell or expose over MCP. The --project, --memory-type, and --recency flags are what make that narrow niche practical for real workflows instead of just toy examples.

# In your CLAUDE.md or agent instructions:
# Use `vipune search` before starting work to check for relevant context.
# Use `vipune add` to store decisions, discoveries, and guardrails.
# Use `vipune add --memory-type guard` for things that must not be forgotten.

That is the whole integration.

GitHub | crates.io | CLI Reference