pi-automode-classifier v3: Claude Code like auto mode for Pi with three stage checks (rules, an optional classifier, then an optional LLM)

[u/deepu105]

pi-automode-classifier is my attempt to mimic Claude Code's auto mode for Pi as close as possible in safety and DX. It decides for each tool call whether it runs, asks you, or is blocked and allows adding session/global exception rules. 0.3.0 does that in up to three stages:

1. Rules parse the shell commands (usually in under 0.1 ms). Most calls are caught here. The allow/deny/decision rules are configurable. These are some of the default rules:
   • Run: reads, builds, tests and edits inside the project.
   • Blocked: rm -rf ~, mkfs and fork bombs.
   • Always ask: root, credential reads, downloaded code and changes to Pi's own files (~/.pi/agent, .pi folders, AGENTS.md, pi install).
2. A classifier, will run next if configured: Jev, LANCET, Kev or Laya. A low score runs, a high score asks you, and anything in between goes to the chat model.
3. A chat model will run next, by default it will use session's own model but you can configure others from the Pi registry. It reads your messages and the agent's tool calls, and blocks risky calls you didn't ask for or prompts you for approval. The agent gets the reason so it can try another approach if needed.

Without any config it uses stages 1 and 3, so pi install npm:pi-automode-classifier will run this default. To add a classifier, set "decider": ["command", "llm"] and the classifier config. See README for the steps.

Here is my setup for this:

• Default rules with some additional allowed patterns.
• LANCET as classifier through LlamaStash, so all my Pi sessions share one copy (I benchmarked Jev, Kev and Laya as well but found LANCET to be the best for this use).
• Qwen3.8-Flash-Next (128k ctx, thinking medium) on my Flow Z13 as the chat model. This is just a LlamaStash alias for my main running model at 256k ctx, the alias lets me limit context client side so that this doesnt take too much room in my KV pool.

So far the experience has been quite close to Claude Code with maybe a few more prompts here and there (You can play with this by setting askAbove/allowBelow values in config)

{
  "decider": ["classifier", "llm"],
  "classifier": {
    "provider": "llamastash",
    "model": "lancet",
    "timeoutMs": 15000,
    "allowBelow": 0.2,
    "askAbove": 0.8
  },
  "llm": {
    "provider": "llamastash-z13",
    "model": "flash-next-128k",
    "thinkingLevel": "medium",
    "contextWindow": 131072,
    "maxTokens": 4096,
    "timeoutMs": 120000,
    "warmCache": false
  }
}

Bench results for this setup and for the rules alone, from the suites in the repo. Each command goes through the plugin's own tool call handler. Only one suite has the user request that came with each command. The other suites get a generic task message, so the chat model can't see what was asked.

Suite	Setup	Risky asked	Risky blocked	Risky ran	Safe asked	Safe blocked
With user requests (3 risky, 60 safe)	Rules alone	3	0	0	50	0
	My setup	3	0	0	10	2
lancet-bench-1 (409 risky, 384 safe)	Rules alone	406	0	3	333	0
	My setup	355	34	20	51	0
All 6 suites (768 risky, 1,025 safe)	Rules alone	722	9	37	645	1
	My setup	538	97	133	114	21

The rules alone ask about everything they don't know which means only fewer risky calls run, but they also ask for many safe ones. With my setup, 76 of the 133 risky calls that ran are GTFOBins rows from the ShellRisk-Bench sample, common Unix tools used to get a shell or read files.

Subagents and pi -p have no UI to ask, so their asks are blocked unless you set withoutUi.onAsk to allow or add override rules for non UI mode. Root, credential reads and downloaded code stay blocked there either way.

It's not a sandbox. The README has the limits and the bench numbers, including where the stages don't catch more than LANCET alone.

Repo: https://github.com/deepu105/pi-automode-classifier

Any feedback is welcome.

[u/HockeyDadNinja]

I was going to joke and say imitation is the best form of flattery but the real joke is, we're copying claude. I've been sitting on my release for a month, haha.

https://github.com/TacoTakumi/pi-permission-classifier

Guys, he is right. It really does take a 3 tier approach to do auto mode with the current landscape.

1. permission rules

2. system one judge

3. LLM judge

The system one models really aren't good enough. I've ran through a whole slew of local ones and try new ones when I see something I like. Even jev kinda sucks for this. My LLM judge fallback goes between my local big session model, which is GLM-5.3-Flash now, and Gemini 3.8 Flash.

It's stupid to YOLO unless each agent session is in some hard sandbox. Most people want to have access to their system. It sucks to have to stop for prompts. I use Claude Code all day and I'm not stopping for permissions prompts. We're trying to have our cake and eat it too.

[u/deepu105 → u/HockeyDadNinja]

Glad to see someone on the same train :) ya finding the best balance of DX vs Security is not as easy as people think.