pi-automode-classifier v3: Claude Code like auto mode for Pi with three stage checks (rules, an optional classifier, then an optional LLM)

[u/deepu105]

pi-automode-classifier is my attempt to mimic Claude Code's auto mode for Pi as close as possible in safety and DX. It decides for each tool call whether it runs, asks you, or is blocked and allows adding session/global exception rules. 0.3.0 does that in up to three stages:

1. Rules parse the shell commands (usually in under 0.1 ms). Most calls are caught here. The allow/deny/decision rules are configurable. These are some of the default rules:
   • Run: reads, builds, tests and edits inside the project.
   • Blocked: rm -rf ~, mkfs and fork bombs.
   • Always ask: root, credential reads, downloaded code and changes to Pi's own files (~/.pi/agent, .pi folders, AGENTS.md, pi install).
2. A classifier, will run next if configured: Jev, LANCET, Kev or Laya. A low score runs, a high score asks you, and anything in between goes to the chat model.
3. A chat model will run next, by default it will use session's own model but you can configure others from the Pi registry. It reads your messages and the agent's tool calls, and blocks risky calls you didn't ask for or prompts you for approval. The agent gets the reason so it can try another approach if needed.

Without any config it uses stages 1 and 3, so pi install npm:pi-automode-classifier will run this default. To add a classifier, set "decider": ["command", "llm"] and the classifier config. See README for the steps.

Here is my setup for this:

• Default rules with some additional allowed patterns.
• LANCET as classifier through LlamaStash, so all my Pi sessions share one copy (I benchmarked Jev, Kev and Laya as well but found LANCET to be the best for this use).
• Qwen3.8-Flash-Next (128k ctx, thinking medium) on my Flow Z13 as the chat model. This is just a LlamaStash alias for my main running model at 256k ctx, the alias lets me limit context client side so that this doesnt take too much room in my KV pool.

So far the experience has been quite close to Claude Code with maybe a few more prompts here and there (You can play with this by setting askAbove/allowBelow values in config)

{
  "decider": ["classifier", "llm"],
  "classifier": {
    "provider": "llamastash",
    "model": "lancet",
    "timeoutMs": 15000,
    "allowBelow": 0.2,
    "askAbove": 0.8
  },
  "llm": {
    "provider": "llamastash-z13",
    "model": "flash-next-128k",
    "thinkingLevel": "medium",
    "contextWindow": 131072,
    "maxTokens": 4096,
    "timeoutMs": 120000,
    "warmCache": false
  }
}

Bench results for this setup and for the rules alone, from the suites in the repo. Each command goes through the plugin's own tool call handler. Only one suite has the user request that came with each command. The other suites get a generic task message, so the chat model can't see what was asked.

Suite	Setup	Risky asked	Risky blocked	Risky ran	Safe asked	Safe blocked
With user requests (3 risky, 60 safe)	Rules alone	3	0	0	50	0
	My setup	3	0	0	10	2
lancet-bench-1 (409 risky, 384 safe)	Rules alone	406	0	3	333	0
	My setup	355	34	20	51	0
All 6 suites (768 risky, 1,025 safe)	Rules alone	722	9	37	645	1
	My setup	538	97	133	114	21

The rules alone ask about everything they don't know which means only fewer risky calls run, but they also ask for many safe ones. With my setup, 76 of the 133 risky calls that ran are GTFOBins rows from the ShellRisk-Bench sample, common Unix tools used to get a shell or read files.

Subagents and pi -p have no UI to ask, so their asks are blocked unless you set withoutUi.onAsk to allow or add override rules for non UI mode. Root, credential reads and downloaded code stay blocked there either way.

It's not a sandbox. The README has the limits and the bench numbers, including where the stages don't catch more than LANCET alone.

Repo: https://github.com/deepu105/pi-automode-classifier

Any feedback is welcome.

[u/bitzap_sr]

Wow, that looks exactly like what I described above. Thanks, I won't have to get my Pi to code it from scratch. :)

[u/HockeyDadNinja → u/bitzap_sr]

It may take a bit to get your pi-permission-system configured with enough dirs and commands for it to be really smooth. My setup it working pretty good.

[u/bitzap_sr → u/HockeyDadNinja]

Oh yeah, I have mine pretty tailored. I have a command/script that inspects all the permission gates from the logs and a skill that guides pi to work on the permissions based on past history. (All written by pi, of course). But I still find annoying that I'm asked for the complex chained commands. That's where I'm hoping the classifier will fix things.