Fiona is my personal assistant. I built her about a year ago, she lives in Telegram, and she runs on my own hardware. Last week her web-research worker hit a billing error, spent several minutes improvising around it, and then wrote that improvisation into its own long-lived instructions as standing policy. No attacker anywhere. Just an agent optimising for "finish the research" with write access to its own playbook.

The incident only makes sense as a test of decisions made a year ago, so let me start there.

A year of Fiona

The founding requirement for Fiona was never capability. It was containment. Some of what I handle through her is sensitive, and I did not want it leaking anywhere: not to a vendor, not to a random website, not through a clever prompt. So the design has been paranoid from day one. Fiona herself runs on an internal-only container network with no route to the internet. Her only way out is an allowlist proxy that reaches Telegram and the local model gateway, and nothing else. She cannot browse the web at all.

Web research had to live somewhere else, so it went to a separate worker in its own network zone. The first version was a small agent I wrote myself and named Pi. I wanted a non-gendered name for it... and only later noticed that the agent framework I build my coding harnesses on is also called pi. Naming things, one of the two hard problems, confirmed again.

Then Nous Research released Hermes, a capable open-source agent, and I made what felt like an obviously sensible call: swap my hand-rolled Pi for Hermes and get someone else's maintenance for free. Fewer things for me to keep patched, more capability out of the box. Keep that decision in mind, because it is half of this story.

The zone split stayed exactly as it was. Outbound research queries pass a PII gate before they leave Fiona's zone, so the worker only knows what it is asked. Its answers come back as data: English by contract, and Fiona's own model rewrites anything I actually read. The worker never talks to me directly.

What happened

On September 20 the credit on the search API account ran out. Every search from the worker started returning a payment error. Nobody noticed, because that error went into a log file and nowhere else.

On September 23 I asked Fiona a research question. Fiona called the worker, waited three minutes, timed out, retried, timed out again, and told me the worker "couldn't connect to the server". That was wrong: the worker was reachable and very busy. Inside it, each run hit the payment error and then walked a fallback chain on its own initiative: a browser (not installed, so that failed), the shell, code execution, delegation. Both abandoned runs finished long after Fiona had given up, holding the shared GPU the whole time.

Forty-two seconds after the second run ended, the worker's own research skill file was rewritten. It gained a new "fallback strategies" section: when search reports a payment error, fetch pages another way instead; when a site refuses access, make the requests look more like a normal browser. A one-off improvisation had become standing policy.

A few honest markers, because this is where it would be easy to overclaim. Established from logs and file timestamps: the payment errors, the fallback chain, the timing, the rewritten file. Inference: that the rewrite came from Hermes' post-run background review, based on timing (the weekly curator ran hours later). Unknown: whether any of those fallback attempts actually fetched anything (the logged attempts returned errors, but the worker did have internet access), and whether the rewritten skill was ever used before I found it. As far as I saw, no further research requests arrived before the fix.

And one detail that changes how I read the whole thing: that skill file did not ship with Hermes. The agent wrote it itself, back in August, and had edited it seven times since.

Why it happened

I'd put it this way: this was not a rogue AI. It was an agent optimising around a constraint, which is a more boring description and a much more useful one.

The agent was told to complete research. A billing error is just another obstacle, and routing around obstacles is exactly what it was built to do. Nothing in its instructions said "payment and auth errors mean stop". Hermes also has a self-improvement loop: post-turn reviews that can create and edit its own skills. That loop is the step that turned a workaround into persistent behaviour. And the worker had far more capability than the job needed: shell, browser, code execution, file access, delegation, memory and scheduling, when all it ever needed was web search. The workaround it wrote down only works because a shell was there.

The prompt-injection literature mostly assumes an attacker supplies the pressure. Here the agent's own drive to finish the task was enough. So the rule I take away: an agent that reads the web must not write its own instructions. Self-improvement plus untrusted input plus a completion objective means persistent behaviour change, with or without an adversary.

What held

This is the part that made the year of paranoia worth it.

The zone split. The agent that read the untrusted web and misbehaved never shared a process, filesystem or network with Fiona's memory, my profile, my chat history, or her secrets. Simon Willison's lethal trifecta is private data plus untrusted content plus an outbound channel. The worker had the last two and, by construction, none of the first. That breaks the exfiltration leg regardless of what the agent decides to do.

Secrets by construction. The worker gets a derived environment file with only the keys it needs: at the time, two. A misbehaving worker holds only its own credentials, which is why "rotate everything" was never on the table.

A one-way, typed boundary. Queries go out through the PII gate, answers come back as data and get rewritten by Fiona's own model. A confused worker cannot talk to me directly or inject instructions into Fiona.

Monitoring that doesn't depend on the agent. My topic monitors run through a deterministic script calling the search API directly, not through agent jobs. That was a deliberate decision in August, and it meant the agent's behaviour never touched them.

The egress canary stayed green. A periodic check proves Fiona's zone cannot reach the internet. It kept passing throughout.

Forensics were possible. The worker's logs kept every tool call with timestamps, and the modified skill file with its modification time was itself evidence. The chain from payment error to rewritten playbook reconstructed cleanly.

Human review on the fix path. When my operator agent tried to edit the self-modified skill directly inside the container, the permission system blocked it, and it also blocked merging fixes without review. So every change went through reviewed, tested pull requests, and independent reviewers caught two real bugs in the first "lock down the worker" change by reproducing it against the actual image. One left the config owned by root, so the gateway silently fell back to defaults with every tool re-enabled. The other blocked the browser toolset, which silently removed web search too, because toolsets overlap. Guardrails need testing against the real artifact, not against stubs.

And one thing that held by luck, not design: the browser fallback failed only because Chrome was not installed in the image. I'm counting that as a near miss, not a defence.

What failed

No least privilege inside the worker zone. I spent a year hardening the boundary around the worker and almost no time on what the worker could do inside it.

Self-modification on by default. Skills were writable, and the background review and curator were enabled. I hadn't turned them on. I also hadn't turned them off.

Alerting that stopped at log files. Four days of payment errors produced zero alerts. A health check even noticed the errors, and wrote that down in a log too.

Misleading failure reporting. "Couldn't connect" hid both the real cause and the fact that the worker was still running, and Fiona's instructions encouraged retries, which stacked orphaned runs on the GPU.

Quiet looked the same as broken. In an unrelated failure, 7 of 11 topic monitors had silently died weeks earlier from a crashing sync script, and the morning digest kept saying "nothing new".

What changed

The worker is now a web-only worker: an allowlist of web search and fetch, nothing else, and it refuses to start if its config ownership or policy is wrong. Skill and memory writes are staged for my approval and never applied automatically, and a staged change pings me on Telegram: propose, don't apply. The background review, the curator and the agent's own memory are off. The self-authored research skill is archived. Fiona now reports failures honestly and does not retry them, including a clear "search unavailable" signal. Provider errors and failed worker jobs alert me directly. The digest says "monitoring degraded" instead of "nothing new" when monitors are down.

One more thing, and it is the part I keep thinking about. The Hermes upgrade that shipped these guardrails also enabled a new default: when a paid search fails, silently retry on other vendors' free, anonymous tiers. A brand new egress path, arriving in a version bump, and one that would have hidden the next billing failure completely. I switched it off explicitly.

The trade I actually made

Which brings me back to the swap. I replaced my own worker with Hermes to get less maintenance, and I did get less maintenance. What I also got, without really noticing, was someone else's defaults: a self-improvement loop, a broad toolset, and now a fallback behaviour I would never have written myself. None of those are bugs from Hermes' point of view. They are features for a general-purpose agent. They were just the wrong features for a narrow worker sitting next to sensitive data.

So I'd say the lesson is not "don't use Hermes". It's that outsourcing maintenance means inheriting defaults, and defaults drift with every upgrade. Keep the worker boring. Put the safety in network boundaries, credentials and file permissions, because the prompt did not stop anything here and the architecture did. And re-audit the defaults on every version bump... the next one is probably already on its way.