Two weeks in with Feynman, the open source research agent built on the Pi runtime. The short version: rough around the edges, genuinely useful.

What it does

Feynman is a CLI tool. You ask a research question, it dispatches four parallel agents: a researcher pulling from papers and the web, a reviewer running simulated peer critique, a writer producing structured output, and a verifier checking every citation and killing dead links. Every claim in the output links back to a source. That last part matters more than it sounds.

The other standout feature is feynman audit <arxiv-id>: give it a paper and it compares the stated claims against the actual public codebase. How often does published research actually match the code? Turns out, not always.

How I use it

Two patterns have stuck.

The first is gut-feeling verification. I work in software and AI. You accumulate opinions fast, and not all of them survive contact with the literature. Before I put something in writing or stake a position in a conversation, I'll run it through Feynman. Sometimes it confirms what I thought. Sometimes it finds a paper that reframes the question entirely. Either way I come out better informed than I went in.

The second is writing support. When I am drafting something and need a citation I am typically too lazy and go with gut feeling. Feynman changes that for someone like me. Now I ask Feynman. The verifier agent means I am not getting hallucinated references. That is a real change in how much I trust the tool versus how much I trust my own search habits.

The rough edges

It is a young project. The main friction I hit: subagents do not automatically inherit the LLM provider of the main agent. If you are running against anything other than Anthropic, subagents silently fall back, hit a missing API key, and fail. You do not always notice immediately.

Two things cause this. First, Pi does not expand environment variables in agent frontmatter. If an agent file says model: ${OPENAI_MODEL}, Pi reads that as the literal string, not your configured provider. Second, Pi-subagents only looks for agent definitions in specific paths (.agents/ or .pi/agents/). If agents end up anywhere else, Pi falls back to its own bundled definitions, which hardcode anthropic/claude-sonnet-4-6. That is where the missing API key error comes from, even when you have a different provider set up correctly.

The fix is to expand env vars at bootstrap time, before agent files are written to the path Pi actually searches. The file lands with the model string already baked in, not a placeholder. Feynman does not do this yet out of the box.

When subagents fail, Feynman degrades to the main agent or whichever agent has a working pipeline. The system prompt may not be optimised for the task, but the output is still useful. I have a fork that wires this up properly and adds parallel-cli as the search backend. If you run into the same issue, it might save you some time.

Worth it

The project has the right idea about what matters in a research tool: source-grounded output, not plausible-sounding summaries. For a two week old piece of software it is further along than I expected. I will keep using it.

An AI agent deleted a production database this week. The thread is worth reading, but one detail stands out: Cursor's system prompt explicitly said "don't run destructive operations." The agent ran one anyway.

That is not a bug in the agent. It is a misunderstanding of what a system prompt is.

What a system prompt actually is

A system prompt is a suggestion. A nudge. Something the model considers before deciding what to do. Useful for shaping behaviour, setting context, establishing tone. Not useful for preventing a specific action from happening.

The model reads the instruction and decides how to honour it. Most of the time that works fine. But "most of the time" is not a safety guarantee, and the cases where it fails tend to be the expensive ones.

Where enforcement actually lives

The real control surface is permissions: what commands can run, with what arguments, against what resources. That is where you get enforcement, because it operates outside the model's reasoning loop entirely.

The granularity matters. Consider two commands:

git push origin main
git push --force origin main

Same base command. Completely different blast radius. One is routine, expected, recoverable. The other rewrites history and may be irreversible depending on your remote configuration. Treating them as a single "git push" permission, both allowed or both blocked, is exactly the coarse-grained thinking that causes problems.

What you want is argument-level granularity: allowlists with patterns, denylists for destructive variants, confirmation prompts for anything in a grey zone. A real gate, applied at execution time, not in instructions the model gets to interpret.

What CLAUDE.md and AGENTS.md are for

These files sit even further from enforcement than system prompts. They are Markdown the agent might read, might honour, and might weight differently depending on context. Useful for giving an agent operational context about a project. Not a safety mechanism. Treating them as one is the same mistake as trusting the system prompt to stop a destructive operation.

Devcontainers are a different thing entirely

Containers are useful, but they solve a different problem. A devcontainer limits what the agent can break if something goes wrong. It does not control what the agent decides to do. Sandboxing manages blast radius; it does not replace permission gates.

You need both. A container without argument-level permissions still lets an agent force-push inside its allowed remotes. Permissions without a container still let a misbehaving agent affect the host. They are complementary, not alternatives.

What good looks like

Some tools already get this right. OpenCode has had granular, per-agent permissions for a while. Different subagents, different permission scopes, different blast radius caps. It works, and running it that way for the better part of a year has not cost anything in productivity.

The work to get there is tedious but not complicated: map the actual command surface your agents touch, decide what is safe to allow, what needs a confirmation prompt, and what is hard-blocked, on which paths, against which remotes. Nobody enjoys doing this. It is also the only thing standing between your agent and a force push to main on a Friday afternoon, inside a container or not.

I am building a permissions extension for the Pi agent runtime along the same lines. The gate has to live at the command-and-args layer, where execution happens, not in instructions the model is expected to follow.

System prompts steer. Permissions stop.

I keep hearing versions of the same story from consultants and in-house devs alike: "I just used Claude Code to fix that bug." Own API key, own account, company codebase. Wild west.

Does the CTO know? Compliance? Legal?

What's actually happening

When an AI agent "reads" a file or "investigates" a database issue (even with read-only access), that content gets pulled into the context window. From there it makes a round trip to an LLM API. Every turn. Source code, schemas, migration files, logs, error messages with real user data baked in.

This is not a theoretical attack surface. This is the default behaviour, happening right now, in organisations that haven't thought carefully about it. The developer isn't being malicious. They're being productive. That's exactly what makes it hard to catch.

Four questions worth asking before the next sprint

1. Which LLM APIs are actually being used in your codebase right now?

Not which ones you've approved. Which ones are actually running. Personal accounts don't show up in your procurement or security tooling. You won't find them in your firewall logs unless you're specifically looking for the right hostnames. Start by asking your developers directly. You may be surprised.

2. Do the terms of those APIs exclude training on your data?

Most major providers offer enterprise tiers with explicit no-training commitments. Personal accounts are a different story. Read the terms. Some providers are unambiguous; others are not. "Your data is not used for training" in a consumer product FAQ is not a legal guarantee. Get it in writing, in a contract, before your code is in the context window.

3. Where does the data land: EU, US, somewhere else entirely?

Data residency matters under GDPR, and increasingly under other frameworks too. A Finnish developer chasing a bug with a personal API key may be routing company data through inference infrastructure in a jurisdiction your DPA has never heard of. Saying "we didn't know" is not a GDPR defence. Saying "it was just a developer tool" is not a GDPR defence either.

4. What happens when an agent pulls PII into context while chasing a user-reported bug?

This one deserves to be said plainly. A developer gets a bug report: "user ID 48291 sees the wrong balance." They hand it to their agent. The agent reads the relevant database query, checks the logs, pulls a sample row to understand the schema. That sample row may contain a real name, a real address, a real transaction history. It is now in the context window of an API running on infrastructure you don't control, under terms you haven't reviewed, in a jurisdiction you may not know.

GDPR doesn't care that it was a quick fix.

The answer is not to ban agentic tooling

That ship has sailed, and it was a good ship. Agentic development workflows are genuinely useful. Trying to prohibit them will just push usage further underground.

The answer is to own your LLM APIs. At the very least, understand how the technology works and buy tokens from a provider whose terms, data residency, and security posture actually match your compliance requirements. Enterprise agreements with the major providers are not expensive relative to the risk. They give you audit trails, data processing agreements, and no-training commitments in writing.

Set up a company account. Give your developers access. Make it easier to do the right thing than to reach for a personal key.

Where a lot of organisations are right now

Half their codebase round-tripping through three different inference providers on personal accounts. No audit trail. No DPA. No idea.

It's unfathomable. And yet, exactly where a lot of organisations are right now.

BYOT is not a developer problem. It's a leadership problem. The people setting engineering culture and tooling policy need to get ahead of this. Before the compliance team finds out the hard way.

First post. The log is open.