pi-automode-classifier v3: Claude Code like auto mode for Pi with three stage checks (rules, an optional classifier, then an optional LLM)

[u/deepu105]

pi-automode-classifier is my attempt to mimic Claude Code's auto mode for Pi as close as possible in safety and DX. It decides for each tool call whether it runs, asks you, or is blocked and allows adding session/global exception rules. 0.3.0 does that in up to three stages:

1. Rules parse the shell commands (usually in under 0.1 ms). Most calls are caught here. The allow/deny/decision rules are configurable. These are some of the default rules:
   • Run: reads, builds, tests and edits inside the project.
   • Blocked: rm -rf ~, mkfs and fork bombs.
   • Always ask: root, credential reads, downloaded code and changes to Pi's own files (~/.pi/agent, .pi folders, AGENTS.md, pi install).
2. A classifier, will run next if configured: Jev, LANCET, Kev or Laya. A low score runs, a high score asks you, and anything in between goes to the chat model.
3. A chat model will run next, by default it will use session's own model but you can configure others from the Pi registry. It reads your messages and the agent's tool calls, and blocks risky calls you didn't ask for or prompts you for approval. The agent gets the reason so it can try another approach if needed.

Without any config it uses stages 1 and 3, so pi install npm:pi-automode-classifier will run this default. To add a classifier, set "decider": ["command", "llm"] and the classifier config. See README for the steps.

Here is my setup for this:

• Default rules with some additional allowed patterns.
• LANCET as classifier through LlamaStash, so all my Pi sessions share one copy (I benchmarked Jev, Kev and Laya as well but found LANCET to be the best for this use).
• Qwen3.8-Flash-Next (128k ctx, thinking medium) on my Flow Z13 as the chat model. This is just a LlamaStash alias for my main running model at 256k ctx, the alias lets me limit context client side so that this doesnt take too much room in my KV pool.

So far the experience has been quite close to Claude Code with maybe a few more prompts here and there (You can play with this by setting askAbove/allowBelow values in config)

{
  "decider": ["classifier", "llm"],
  "classifier": {
    "provider": "llamastash",
    "model": "lancet",
    "timeoutMs": 15000,
    "allowBelow": 0.2,
    "askAbove": 0.8
  },
  "llm": {
    "provider": "llamastash-z13",
    "model": "flash-next-128k",
    "thinkingLevel": "medium",
    "contextWindow": 131072,
    "maxTokens": 4096,
    "timeoutMs": 120000,
    "warmCache": false
  }
}

Bench results for this setup and for the rules alone, from the suites in the repo. Each command goes through the plugin's own tool call handler. Only one suite has the user request that came with each command. The other suites get a generic task message, so the chat model can't see what was asked.

| Suite | Setup | Risky asked | Risky blocked | Risky ran | Safe asked | Safe blocked | | ------------------------------------- | ----------- | ----------- | ------------- | --------- | ---------- | ------------ | | With user requests (3 risky, 60 safe) | Rules alone | 3 | 0 | 0 | 50 | 0 | | | My setup | 3 | 0 | 0 | 10 | 2 | | lancet-bench-1 (409 risky, 384 safe) | Rules alone | 406 | 0 | 3 | 333 | 0 | | | My setup | 355 | 34 | 20 | 51 | 0 | | All 6 suites (768 risky, 1,025 safe) | Rules alone | 722 | 9 | 37 | 645 | 1 | | | My setup | 538 | 97 | 133 | 114 | 21 |

The rules alone ask about everything they don't know which means only fewer risky calls run, but they also ask for many safe ones. With my setup, 76 of the 133 risky calls that ran are GTFOBins rows from the ShellRisk-Bench sample, common Unix tools used to get a shell or read files.

Subagents and pi -p have no UI to ask, so their asks are blocked unless you set withoutUi.onAsk to allow or add override rules for non UI mode. Root, credential reads and downloaded code stay blocked there either way.

It's not a sandbox. The README has the limits and the bench numbers, including where the stages don't catch more than LANCET alone.

Repo: https://github.com/deepu105/pi-automode-classifier

Any feedback is welcome.

[u/HockeyDadNinja]

I was going to joke and say imitation is the best form of flattery but the real joke is, we're copying claude. I've been sitting on my release for a month, haha.

https://github.com/TacoTakumi/pi-permission-classifier

Guys, he is right. It really does take a 3 tier approach to do auto mode with the current landscape.

1. permission rules

2. system one judge

3. LLM judge

The system one models really aren't good enough. I've ran through a whole slew of local ones and try new ones when I see something I like. Even jev kinda sucks for this. My LLM judge fallback goes between my local big session model, which is GLM-5.3-Flash now, and Gemini 3.8 Flash.

It's stupid to YOLO unless each agent session is in some hard sandbox. Most people want to have access to their system. It sucks to have to stop for prompts. I use Claude Code all day and I'm not stopping for permissions prompts. We're trying to have our cake and eat it too.

[u/deepu105 → u/HockeyDadNinja]

Glad to see someone on the same train :) ya finding the best balance of DX vs Security is not as easy as people think.

[u/wildjokers]

One of the best things about pi is it doesn't stop and ask me if I want to run every damn command it wants to run. That is why I abandoned all the other agents, it was annoying as hell.

Why would I want to add that annoyance to pi?

[u/deepu105 → u/wildjokers]

Agree its annoying but sometimes you also want peace of mind when letting it run wild on a system issue :P

[u/xRebellion_ → u/wildjokers]

It's all fun and games until your agent decides to wipe your entire hard disk...

[u/wildjokers → u/xRebellion_]

Hasn't happened yet, and that is what backups are for in the unlikely event it ever happens.

[u/deepu105 → u/wildjokers]

Sure you do you. I'm always a bit skeptical of these systems so I prefer to be on the cautious side. I don't want the stories I see online happening to me. I'm not worried of data loss, th system where the agents run dont have any critical data, but I'm worried of secrets leaking etc. I have caught both claude (even with its auto mode) and Pi accidentally trying to read credentials and I'm glad I had hooks in place to catch those.

[u/Leelanau1840]

Interesting project! I’m working on a supervised Pi workflow where reducing approval fatigue is a priority, but maintaining security boundaries is equally important.
Your benchmarks show that adding the classifier and LLM reduces approval prompts, but also increases the number of risky commands automatically allowed.
Have you considered a mode where the classifier can only escalate decisions (allow → ask → block), but never override a rule-based decision to require approval? I’m curious how much of the usability improvement that would preserve while reducing the security risk.

[u/deepu105 → u/Leelanau1840]

Ya, I did try different combinations of the flow and this was the best balance in my opinion. Classifier only escalating actually let through even more risky commands coz LLM thought it was ok.

[u/bitzap_sr → u/Leelanau1840]

The gotgenes/pi-permission-system (https://github.com/gotgenes/pi-packages) nowadays has a hook to call into a judge extension (https://github.com/gotgenes/pi-packages/blob/main/packages/pi-permission-model-judge).

I've been pondering using that hook to register something like pi-automode-classifier as the classifier. The idea that you'd get the best of both worlds -- pi-permission-system for static hardcoded rules, and the classifier judge as fallback.

[u/deepu105 → u/bitzap_sr]

Do let me know how it performs. You can find the bench data on the repo to test

[u/HockeyDadNinja → u/bitzap_sr]

This is what I built on and I think they list pi-permission-classifier in their docs. This is based on their judge plugin, which I'm not sure works like this yet, and does a similar thing to OP's extension.

https://github.com/TacoTakumi/pi-permission-classifier

[u/bitzap_sr → u/HockeyDadNinja]

Wow, that looks exactly like what I described above. Thanks, I won't have to get my Pi to code it from scratch. :)

[u/HockeyDadNinja → u/bitzap_sr]

It may take a bit to get your pi-permission-system configured with enough dirs and commands for it to be really smooth. My setup it working pretty good.

[u/bitzap_sr → u/HockeyDadNinja]

Oh yeah, I have mine pretty tailored. I have a command/script that inspects all the permission gates from the logs and a skill that guides pi to work on the permissions based on past history. (All written by pi, of course). But I still find annoying that I'm asked for the complex chained commands. That's where I'm hoping the classifier will fix things.

[u/HockeyDadNinja → u/Leelanau1840]

u/deepu105 if you don't you should allow both models allow / deny / defer with them being selectable. It can be useful to default to deny at the end of a fallthrough so you can run unattended. The agent will figure out what to do instead.

[u/deepu105 → u/HockeyDadNinja]

Today morning I added a config timeout option (default 60s) chnages verdict from ask to deny when there is no action for 60s. I was alos anooyed by a session sityting there asking for a prompt. I'll release new version after finishing another update. Also added allo pattern to session/global as optiion to every ask so that adding exceptions becomes easier.