Token-saving shell filters that do not save tokens, what?

None of the three shell-output filters I tested reduced tokens consistently. Across six tasks in the Flask codebase, six repetitions each, with the same model and the same prompt, the typical run with codemode, oo or rtk in the loop used more tokens than the plain baseline, not fewer. One exception stands out: on the task built to reward output filtering, a single token hidden inside a 165 KB passing test run, oo's built-in pytest summary cut the typical run from 34,948 tokens to 12,096, about a third. Everything else in this post is the detail behind those two sentences, and enough of the method that you can check them.

The problem these tools solve

A coding agent is a language model in a loop: it reads your code, runs shell commands, reads what they print, and decides what to do next. Everything a command prints goes into the model's context, and the model pays for every line of it in tokens. Tokens are the unit you are billed by if you use a hosted model, and the unit that fills up the model's context window either way. A test suite that prints 165 KB to say "all passed" costs the agent tens of thousands of tokens to learn one bit of information. Linter output, git logs, directory listings: most of what the shell says is noise to the model, and it reads all of it.

Three tools try to fix that, from different angles. I built one of them, so read what follows with that in mind. It is also why the method is spelled out to the flag and the harness, the scripts that build the tasks and run and grade the agent, is available to anyone who asks.

oo is mine. It is a wrapper you put in front of a shell command: you run oo cargo test instead of cargo test. oo runs the command, keeps the result and any errors, and files the rest away so the agent can ask for it later with oo recall if it turns out to need it. It knows the output shapes of common tools (test runners, builds, linters) and you can teach it new ones. For pi it installs as an extension that quietly adds the oo prefix to the agent's commands.

rtk is the same shape of idea from a different author: a command prefix, a single Rust binary, with a configurable pipeline that compacts the output of git, ls, test runners and so on. It ships an official pi extension and advertises 60-90% savings on common dev commands.

codemode is different. It is a tool built into pi, the coding agent I use, since pi 1.0: instead of the model calling one tool at a time and reading each result, it writes a small JavaScript program that calls pi's tools from inside a sandbox, and only what that program prints or returns comes back to the model. So a script can run the whole test suite, search the output for the one line it needs, and return that line: the model never sees the rest. The idea is not pi's alone, Cloudflare wrote it up as Code Mode last year and several agents have adopted it since. The filtering is whatever the model chooses to write into its script, which is both the power and the catch. pi has a setting for how hard to push the model towards it: on, where the normal tools stay available alongside codemode, and only, where they are hidden and everything has to go through a script. The codemode docs have the details.

How I tested them

The code under test is Flask, the Python web framework, at one fixed commit (22d92470). A seeded script plants six changes into it, each with a hidden answer key, so anyone on the same commit gets the same six tasks:

• t1-rootcause: dozens of tests fail; find the file and line of the planted bug that causes them.
• t2-infoloss: a single token is printed once inside a 165 KB passing test run; find it. This task is built for output filters to shine on.
• t3-riskdiff: a commit touches many lines of whitespace and one line of behaviour; find the one that matters.
• t4-history: which commit, in a long history, last changed a given behaviour.
• t5-lint: fix one real undefined name planted among hundreds of lines of linter noise; ruff plus tests.
• t6-control: find one value in a 3,000-line file, where no filter should help. The answer key is a planted canary string, not a credential.

The agent is pi 1.1.0, run without its interactive screen, one task per run, with a flag set that gives every arm the same prompt and the same four basic tools (read, bash, edit, write). The only thing that differs between the six configurations, the arms in benchmark-speak, is what sits between the agent and the shell.

The model is Qwen3.8-27B in a 4-bit INT4 format, served by vLLM from the Trail Openers H100, our team's shared production GPU (the H100 post has the setup), with the model's thinking mode off. That endpoint has no per-token price, so I count tokens, not money.

Six arms:

• plain: pi's four basic tools, nothing in between. The baseline every ratio is measured against.
• codemode: pi's built-in codemode tool switched on in its default on mode; the normal tools stay available and the model may write a script if it wants to.
• codemode-only: codemode in only mode, the normal tools hidden so every action goes through a script. This is the test of the mechanism itself.
• oo: a pi extension puts oo in front of the agent's simple shell commands. oo's built-in output patterns only.
• oo-patterned: as oo, plus one extra pattern I wrote that teaches oo the summary line of the task's own test script.
• rtk: the official rtk pi extension, as it ships in rtk 1.0.1, nothing tuned.

Versions, for anyone rerunning: oo is double-o 0.10.1 (cargo install double-o --version 0.10.1 --locked; the package is called double-o, the command is oo). rtk is 1.0.1, released 2026-10-14. Two runs at a time on the shared GPU, 15 minutes per run before it counts as a timeout, every run on a fresh copy of its task repo with its own oo and rtk state. Grading is strict: the exact file and line, or the exact string, or it is a fail. Timeouts are fails and stay in the data.

Results

216 runs, six per task per arm. "Median" below means the middle one of the six.

The last column is a median of six per-task medians, which means one task can swing it, so do not stop at this table. Median tokens per task, with the ratio to plain in brackets (under 1.00 means fewer tokens than plain):

What I read in that table.

The agent's luck swamps the filters. Look at t1: plain pi used anywhere from 33,757 to 1,038,931 tokens across six runs of the same task. rtk, from 24,413 to 11,429,960. A factor of thirty or more between the best and worst run of the same thing. What decides a run is whether the agent happens to look at the planted change early, and no output filter decides that. On the transcripts from the first rtk version I tested, every run that displayed the planted commit got the answer right, and every run that never displayed it answered from its memory of upstream Flask and got the line wrong. I have not re-checked that on the final rtk transcripts, so I will not claim a cause for rtk's t1 median of 40 turns against plain's 12.

t2 is the one place a filter did what it says on the tin. 165 KB of pytest output with the answer printed once inside it. oo's built-in pytest pattern collapsed that run to a one-line summary, the agent asked oo recall for the detail it needed and found the token: 12,096 tokens, 6 of 6 correct, three turns. That is the design working as intended. And the honest footnote: the oo-patterned arm, where I added a hand-written pattern for the task's own test script, did worse on t2 than plain oo, 26,439 against 12,096. My pattern was not better than the built-in one. codemode-only also helped here (0.73), rtk did not (1.07).

oo-patterned is the only arm that got everything right, 36 of 36, and the only one whose overall ratio is under one. I am not going to present 0.93 as a general saving: it is one task's effect spread across a median. What I'd take from it is the pass rate. Fewer tokens is the pitch, but an agent that reads a summary instead of 165 KB of noise also seems to get fewer things wrong, and I'd like to know whether that holds at a bigger sample.

codemode-only timed out five times, each after millions of tokens, stuck in a loop: the script fails, the model rewrites it, it fails again. With codemode merely available alongside the normal tools (the codemode arm), this model used it in one run out of 36 and otherwise ignored it. So codemode as pi ships it was not really exercised here; codemode as a forced mechanism was, and the result cuts both ways. On t1 and t5 it was the cheapest arm by far, a quarter and half of plain, because a script that runs the tests and searches the output is exactly what those tasks want. On t3, t4 and t6 it was the most expensive, because the model wrote scripts for work that was one git log away. The idea is sound, and I'd say the open question is whether the model knows when to reach for it. This one, with thinking off, did not.

rtk at its defaults cost tokens on every task, none of them dramatically apart from t1, where the spread makes the median unreliable anyway. Its own counters show 463 commands rewritten across 36 runs, so the filter was in the loop. I did not tune its pipeline, and its file-reading command defaults to full content, so this is rtk as installed, not rtk as configured by someone who knows it well.

The one concrete thing rtk's filter dropped

On t5, the lint task, the agent runs ruff, a Python linter that can also fix what it finds. Plain pi sees ruff's own summary: "Found 301 errors (300 fixed, 1 remaining)". In the rtk runs, the agent sees "Ruff: 1 issues in 1 files" and the one remaining error. The part saying that ruff had just auto-fixed 300 things, by deleting 300 unused imports, is gone. Confirmed on both rtk versions I ran.

In these runs it did not change the outcome: the planted defect was the one remaining error, and every rtk run found it. The agent simply never learned that its first ruff call had rewritten the file under it. I'd put it this way: that is exactly the kind of line a filter has to make a decision about, and rtk's default decided wrong for this case. rtk's maintainers have not seen this setup; I have not sent it to them.

Two smaller observations I will not build on: rtk's compacted git diff drops the header lines (diff --git, index) that an agent might search for, though no run here was derailed by it. And one rtk directory listing said "3 filtered" where one entry was hidden. One run, unexplained.

Failures, all 13

• t1: rtk rep 2 answered line 635 (key 636) after 615,816 tokens and 53 turns. oo rep 4 answered 637. oo rep 1 timed out at 2,794,061 tokens. plain rep 1 answered 641 after 1,038,931 tokens. codemode rep 6 answered 635. codemode-only rep 3 timed out at 17,405,554 tokens.
• t3: codemode-only rep 2 timed out at 25,425,344 tokens; rep 4 answered 194 (key 193).
• t4: codemode-only rep 3 timed out at 21,114,303 tokens.
• t5: plain rep 3 timed out at 97,927 tokens.
• t6: codemode-only reps 2 and 5 timed out at 36,034,105 and 16,739,778 tokens.

Several of the t1 misses are one line off. They count as fails under the strict rule, and I have not computed a lenient count.

Things that went wrong on the way

The first pass had a leak. pi's system prompt includes its own install path, and I had installed pi inside the project folder, so five runs read that path and wrote their answer into the harness folder instead of their task repo. pi and the test environment moved out of the tree, all 216 runs were redone, and no final transcript writes outside its own repo. The first-pass rows are archived, not deleted.

oo changed under me: the first runs were on a development build, the final runs are on the released 0.10.1, which fixes what oo recall can see. rtk was first run at 0.51.0, then rerun at 1.0.1 when it shipped; the 1.0.1 rows are the result reported here, and both versions rewrite the same commands.

And the limits: one model, one repo, six synthetic tasks, six repetitions each, on a shared GPU two runs at a time. Enough to say "not consistent", not enough to rank the three.

Rerun it

The harness is not public yet, but it is available on request: a README with the step-by-step, the seeded script that builds the six tasks from the Flask commit, the 216 result rows as JSONL and the script that regenerates the report from them. Everything is pinned (pi 1.1.0, double-o 0.10.1, rtk 1.0.1, Flask at 22d92470); you supply your own OpenAI-compatible model endpoint. If you run it with a different model, a tuned rtk pipeline, or a codemode arm that forces scripting from the first turn, I want to see the table...