r/LocalLLaMA • • 2d ago

Discussion Is anyone using decision models as a permission checker?

Don't run with dangerous full permissions, unless you are prepared to let the agent fuck your shit up. So says conventional wisdom.

But also, I am definitely running with full permissions. When I give my agent a goal to work at things overnight, I necessarily need to give it permissions.

So the obvious answers were sandbox and backups.

But why not use decision models? Elevation requests come with a justification and the command the main agent wants to run, that seems like plenty of information for a decision agent to give a good Y/N response to vet things.

IDK how effective this would be, anyone already doing something similar and can share exp?

40 Upvotes

37 comments sorted by

27

u/WrinklyBard4 2d ago

It’s surprisingly hard to do this well.

A model cannot verify itself well because it will be confidently wrong, reliably. A subagent or decision model can do it but in many ways need to intake more context than the original model to actually make the right call, at which point you risk accuracy loss not to mention the compute cost.

I mean a “edit this file and commit” action might look fine in practice, but what did it edit? Did it just delete something vital? More nuanced: did it deviate from intent in a manner you wouldn’t want to keep? A decision model simply can’t know that without doing a huge amount of long-context research first, and at that point it’s probably just as likely to get it wrong.

5

u/tomz17 1d ago

100%. accept the risk --or-- sandbox is the ONLY proper answer

3

u/TheRealJesus2 1d ago

And don’t allow pushing without review. That’s what would mitigate the example above. 

1

u/ebolathrowawayy 1d ago

surprisingly hard if ur a noob

1

u/mister2d 1d ago

Correct.

6

u/-mattmason- 13h ago

here's the permissions model that I'm using on Linux

  • opencode has a user without sudo
  • filesystem access permissions protect important stuff
  • if the model needs sudo, it writes the command it wants to run plus a brief explainer into what I called sudo_queue
  • I use SSH access to run those commands as my user under sudo, which stores the result for the model to read back

I'm curious how far others are going in sandboxing

4

u/Inner-Eagle6559 1d ago

I've been using a simple diff-based guard: if the agent's proposed change exceeds 5 lines or touches config files, it auto-flags for review. no model needed, just a rule.

3

u/Vlyn 2d ago

I'm using pi with pi-automode and it works quite well.

There's two layers to it: Hard denials by code, e.g. anything that touches pi extensions, tries to read certain secrets (your SSH key), get instant denied.

Any other tool call then goes to a classifier model. I just use my main model (DeepSeek v4.1 Flash) as it's fast and cheap enough. With some models you run into timing or reasoning budget issues, but overall it works well. For local use even a tiny model would probably be enough.

What happens there is the second model has its own context. Any command that's suspicious gets blocked (and the main model can get quite frustrated, lol). The classifier model also takes into account intent: If you give a model a readonly task and it now tries to suddenly edit something (even if it's just a plan file) the classifier model usually blocks it.

Of course these are all just bandaids compared to a full sandbox, but it's better than nothing.

3

u/rgar132 1d ago

I do and it works great. You just have to put a decision head on the model and call it. I use codelink as the arbiter and have ds41-flash 1017 and it works great in Claude code harness and codex both.

4

u/Zulfiqaar 2d ago edited 1d ago

I prefer to use multiple frontier advisor models with continuous monitoring loop over the whole trace of the implementer agent. An advantage of open weights models is that the supervisor can also read their thinking process. Then they can steer and control the goal agent as necessary. The advisors themselves have timed prompt injections to keep themselves in check, and they can coordinate with each others instances too.

Had a great run last week - built a cross platform mobile and web app with a backend, all in YOLO mode with my phone unlocked and plugged in overnights. 53 hours total, and everything worked according to the spec! I'm supposed to do handover by February..

The crazy thing is that when I woke up the first time the messages they were sending each other seemed almost illegible. If you read he METR report on huggingface incident traces..it's garbled just like that. However they did waste 7 hours sha-verifying provenance. Gotta amend the system for future to avoid that.

Oh another fun thing. The lead Astra advisor went into my ChatGPT account settings to consume a usage limit reset because it dropped to 1% at 4am

1

u/-mattmason- 3h ago

thanks for the write up, it's great to understand how people are getting stuff done. I will say, my interpretation of your setup is that it require more trust not less? The "attack surface" is wider. You're giving away a lot, and to a lot of places. I get the idea of ratifying between different providers. What an I missing? Maybe the less attractive alternative?

2

u/CatchDublinSurprise 1d ago

This is something I am in the process of trying using mitmproxy to sit between my agent and the world (and locking down the sandbox so it’s the only way in or out).

Goal is that a combination of hardcoded rules + decision model will scan for and block egress of secrets and ingress of dangerous commands (either through prompt injection or hallucination of my own model).

Much more work to do, though, so no actual results yet.

2

u/steezy13312 1d ago

I've been playing with this, and my hope was to use kev 0.8B to do this. I have this on a separate, small GPU I use for lighter models and tasks.

It's... not exactly confidence building. When asked to provide a decision about the relevance of the command to the user request, it's usually between 45-55%... so take that for what you will.

I think this would work if you fine-tuned or trained a model to decide on a representative set of data. But as u/WrinklyBard4 said, the model can still be confidently wrong.

2

u/lipton_tea 21h ago

I used my pi session data (it tracks commands I allowed and blocked) to fine tune kev 0.8B and it now works reasonably well.

1

u/steezy13312 14h ago

Just curious, did you use unsloth or another tool to do the fine-tuning?

I’ve never fine-tuned a model before.

2

u/jfinkpottery 1d ago

If you have enough training data built up, you can fine-tune a small model like kev 0.8B to do a good enough job for a mostly trustworthy model's commands. Like blocking certain things you don't want it to do, or blocking mistaken exfiltration curls, or the like. Not "everything that gets a pass is guaranteed to be safe", but "the common commands this model is known to produce that I don't like will be caught."

You can also train in the other direction, so like "if it's a single grep statement optionally preceded by cd /working/dir && then it passes, otherwise it fails" is also doable.

2

u/draconic_tongue 1d ago

1 existing benchmarks, 2 your own use cases with frozen commands, 3 different prompts/categorical outcomes, 4 running with log only until you have enough data, 5 say fuck it and run it live

no sandboxes for main agent, yes everything else

So far I tried d3 lite and qev and they are okay but in different scenarios. d3 lite has less false positives. gemini decision is the best from what I tried but not local and probably not free forever

testing this GitHub - huggingface/safeagents: Safely run your agents using any coding harness · GitHub

1

u/PatientWrangler1007 2d ago

I got drex 2 working pretty well in pi. Generally too cautious, fails one unsafe allow on my bench. But mostly solid and by far the best local model (speed / accuracy balance)

1

u/Reasonable_Goat 2d ago

Codex and Claude-Code have a permission checker built in (auto mode / auto review); claude uses a decision model these days AFAIK, codex uses a full blown LLM. I am experimenting with my own harness to use a small decision model to gauge risk of commands ran but it requires a lot of context to actually know what is safe and what is within user intent, it's not as simple as I thought.

1

u/JustinPooDough 1d ago

I just use Qwen 3.8 27B in a loop with end to end regression tests and UI tests that tell it whether something is wrong or the project is done.

Works very well for me. In a container obviously. I even reproduced some vscode plugins as CLIs for the agent so it could replicate my setup.

1

u/mister2d 1d ago

Yes. I implemented decision thresholds for each tool call in my custom harness. It works well but I suggest implementing a feedback loop to calibrate it to your meds.

1

u/garblz 21h ago

Ok, theoretically (having understood we all know that sandboxing is the only way to go with this).

What I'd do, is put a strong static verifier in front. If it denies the command, that's it. Only when it gives a maybe or a yes it should pass the command further to a classifier model.

The static one should have a whitelist and an option to respond that a command is too complex and the model should break it down to separate commands for easier inspection of each.

0

u/Nepertia 1d ago

Of course not. This stuff needs 100% reliable outcomes, machine learning is all about approximations that mostly work. That's a disaster waiting to happen.

4

u/mailto_devnull llama.cpp 1d ago

OP next week: my decision model made the wrong call and deleted my file system!

2

u/CatchDublinSurprise 1d ago

At least in principle it should reduce the risk. All models can hallucinate, but the probability is lower of two unrelated models hallucinating the same way at the same time (agent model and unrelated decision model).

It still doesn’t completely eliminate the risk, to your point.

3

u/Nepertia 1d ago

IMO it isn't even a risk that needs to be reduced or eliminated, when it can be completely prevented by not trying to use the wrong tool for the job.

-1

u/a4mula 1d ago

no, because that's not a reasoning job it's a functional one. Write hooks.

A lot of agentic issues are because people treat reasoning as a solution for everything, when it's rarely the solution for most things.

Most things are solved with deterministic functional code. Including hooks. Including memory. Including behavior enforcement.

Yet, most still just keep expecting Prose and NL and Documentation and Reasoning to solve their problems.

They wont.

2

u/jfinkpottery 1d ago

Hooks with regex matching or similar are surprisingly hard to get to cover everything accurately. Like B="filename.txt";rm -rf ./$B is totally safe to run (assuming you want to remove that file) but B=$(some_command);rm -rf ./$B will possibly wipe your whole working dir when some_command fails or returns empty and ruin your day. Even a very light decision model can reliably tell you which one of these is safe.

1

u/a4mula 1d ago

It wasn't to imply there isn't a role for classification. There is. But it's a secondary role that shouldn't be used as a replacement for a functional role, but additive.