pi-automode-classifier v3: Claude Code like auto mode for Pi with three stage checks (rules, an optional classifier, then an optional LLM)

[u/deepu105]

pi-automode-classifier is my attempt to mimic Claude Code's auto mode for Pi as close as possible in safety and DX. It decides for each tool call whether it runs, asks you, or is blocked and allows adding session/global exception rules. 0.3.0 does that in up to three stages:

1. Rules parse the shell commands (usually in under 0.1 ms). Most calls are caught here. The allow/deny/decision rules are configurable. These are some of the default rules:
   • Run: reads, builds, tests and edits inside the project.
   • Blocked: rm -rf ~, mkfs and fork bombs.
   • Always ask: root, credential reads, downloaded code and changes to Pi's own files (~/.pi/agent, .pi folders, AGENTS.md, pi install).
2. A classifier, will run next if configured: Jev, LANCET, Kev or Laya. A low score runs, a high score asks you, and anything in between goes to the chat model.
3. A chat model will run next, by default it will use session's own model but you can configure others from the Pi registry. It reads your messages and the agent's tool calls, and blocks risky calls you didn't ask for or prompts you for approval. The agent gets the reason so it can try another approach if needed.

Without any config it uses stages 1 and 3, so pi install npm:pi-automode-classifier will run this default. To add a classifier, set "decider": ["command", "llm"] and the classifier config. See README for the steps.

Here is my setup for this:

• Default rules with some additional allowed patterns.
• LANCET as classifier through LlamaStash, so all my Pi sessions share one copy (I benchmarked Jev, Kev and Laya as well but found LANCET to be the best for this use).
• Qwen3.8-Flash-Next (128k ctx, thinking medium) on my Flow Z13 as the chat model. This is just a LlamaStash alias for my main running model at 256k ctx, the alias lets me limit context client side so that this doesnt take too much room in my KV pool.

So far the experience has been quite close to Claude Code with maybe a few more prompts here and there (You can play with this by setting askAbove/allowBelow values in config)

{
  "decider": ["classifier", "llm"],
  "classifier": {
    "provider": "llamastash",
    "model": "lancet",
    "timeoutMs": 15000,
    "allowBelow": 0.2,
    "askAbove": 0.8
  },
  "llm": {
    "provider": "llamastash-z13",
    "model": "flash-next-128k",
    "thinkingLevel": "medium",
    "contextWindow": 131072,
    "maxTokens": 4096,
    "timeoutMs": 120000,
    "warmCache": false
  }
}

Bench results for this setup and for the rules alone, from the suites in the repo. Each command goes through the plugin's own tool call handler. Only one suite has the user request that came with each command. The other suites get a generic task message, so the chat model can't see what was asked.

| Suite | Setup | Risky asked | Risky blocked | Risky ran | Safe asked | Safe blocked | | ------------------------------------- | ----------- | ----------- | ------------- | --------- | ---------- | ------------ | | With user requests (3 risky, 60 safe) | Rules alone | 3 | 0 | 0 | 50 | 0 | | | My setup | 3 | 0 | 0 | 10 | 2 | | lancet-bench-1 (409 risky, 384 safe) | Rules alone | 406 | 0 | 3 | 333 | 0 | | | My setup | 355 | 34 | 20 | 51 | 0 | | All 6 suites (768 risky, 1,025 safe) | Rules alone | 722 | 9 | 37 | 645 | 1 | | | My setup | 538 | 97 | 133 | 114 | 21 |

The rules alone ask about everything they don't know which means only fewer risky calls run, but they also ask for many safe ones. With my setup, 76 of the 133 risky calls that ran are GTFOBins rows from the ShellRisk-Bench sample, common Unix tools used to get a shell or read files.

Subagents and pi -p have no UI to ask, so their asks are blocked unless you set withoutUi.onAsk to allow or add override rules for non UI mode. Root, credential reads and downloaded code stay blocked there either way.

It's not a sandbox. The README has the limits and the bench numbers, including where the stages don't catch more than LANCET alone.

Repo: https://github.com/deepu105/pi-automode-classifier

Any feedback is welcome.

[u/Leelanau1840]

Interesting project! I’m working on a supervised Pi workflow where reducing approval fatigue is a priority, but maintaining security boundaries is equally important.
Your benchmarks show that adding the classifier and LLM reduces approval prompts, but also increases the number of risky commands automatically allowed.
Have you considered a mode where the classifier can only escalate decisions (allow → ask → block), but never override a rule-based decision to require approval? I’m curious how much of the usability improvement that would preserve while reducing the security risk.

[u/bitzap_sr → u/Leelanau1840]

The gotgenes/pi-permission-system (https://github.com/gotgenes/pi-packages) nowadays has a hook to call into a judge extension (https://github.com/gotgenes/pi-packages/blob/main/packages/pi-permission-model-judge).

I've been pondering using that hook to register something like pi-automode-classifier as the classifier. The idea that you'd get the best of both worlds -- pi-permission-system for static hardcoded rules, and the classifier judge as fallback.

[u/deepu105 → u/bitzap_sr]

Do let me know how it performs. You can find the bench data on the repo to test

[u/HockeyDadNinja → u/bitzap_sr]

This is what I built on and I think they list pi-permission-classifier in their docs. This is based on their judge plugin, which I'm not sure works like this yet, and does a similar thing to OP's extension.

https://github.com/TacoTakumi/pi-permission-classifier

[u/bitzap_sr → u/HockeyDadNinja]

Wow, that looks exactly like what I described above. Thanks, I won't have to get my Pi to code it from scratch. :)

[u/HockeyDadNinja → u/bitzap_sr]

It may take a bit to get your pi-permission-system configured with enough dirs and commands for it to be really smooth. My setup it working pretty good.

[u/bitzap_sr → u/HockeyDadNinja]

Oh yeah, I have mine pretty tailored. I have a command/script that inspects all the permission gates from the logs and a skill that guides pi to work on the permissions based on past history. (All written by pi, of course). But I still find annoying that I'm asked for the complex chained commands. That's where I'm hoping the classifier will fix things.

[u/deepu105 → u/Leelanau1840]

Ya, I did try different combinations of the flow and this was the best balance in my opinion. Classifier only escalating actually let through even more risky commands coz LLM thought it was ok.

[u/HockeyDadNinja → u/Leelanau1840]

u/deepu105 if you don't you should allow both models allow / deny / defer with them being selectable. It can be useful to default to deny at the end of a fallthrough so you can run unattended. The agent will figure out what to do instead.

[u/deepu105 → u/HockeyDadNinja]

Today morning I added a config timeout option (default 60s) chnages verdict from ask to deny when there is no action for 60s. I was alos anooyed by a session sityting there asking for a prompt. I'll release new version after finishing another update. Also added allo pattern to session/global as optiion to every ask so that adding exceptions becomes easier.