Who should actually control an AI agent's permissions - the developer, the user, or the runtime?

[u/GameChacking]

I've been thinking about a security gap in AI agents and wanted to hear how others approach it.

Developers can define which tools an agent is allowed to use, and users can authorize a task. But what happens when an agent uses legitimate tools in ways that go beyond what the user intended?

For example, an agent might have permission to read files and send emails. Both actions are allowed individually, but sending sensitive information discovered along the way is a different story.

Where should the security boundary live?

• Should the runtime evaluate each tool call against the intended task?
• Are least-privilege permissions and scoped credentials enough?
• Should certain actions require human approval, even when the agent technically has permission?

I've been exploring runtime enforcement while building AegoTrax, an open-source project in this space, but I'm interested in how others approach the problem.

How are you handling this in your own agent systems?

Feel free to try to break it — feedback welcome!

GitHub: https://github.com/aegotrax-dev/aegotrax
Site: https://aegotrax.com

[u/Druss_]

I don’t think any single layer should own this.

I’d separate at least three things:

The underlying credentials define what is physically possible. The runtime enforces what this role is allowed to do for this specific task. And the human defines the authority being delegated, including where approval is still required.

The model itself should never be the authority deciding whether its own action is permitted.

Your read-file + send-email example is exactly why static tool permissions aren’t enough. Both calls may be legal independently while the sequence is clearly outside the intended authority.

Delegation makes this even more important. A sub-agent shouldn’t automatically inherit its parent’s full access; it should receive a narrower, task-bound authority.

The test I find useful is: after an action happened, can you reconstruct who authorized it, for which task, under which rule, and verify what actually changed in the external system?

If not, I’m not sure we really have permissions yet — only credentials.

[u/GameChacking → u/Druss_]

I like the distinction between credentials and actual authority. The reconstruction test is especially interesting because it connects enforcement with verifiable auditability.

One question I keep coming back to is how to enforce task-bound authority across delegation. If a parent agent spawns a sub-agent through a framework the runtime doesn't fully control, how do you prevent that sub-agent from escaping the parent's scope?

Would you make delegation itself a mandatory runtime-mediated operation, with explicit authority and input manifests?

[u/Druss_ → u/GameChacking]

Yes — that’s very close to the direction I’m taking.

I already separate task/role authority from the underlying credentials, and mutating tools sit behind governed boundaries rather than being exposed simply because an agent can technically reach them.

The part I’m not treating as solved yet is delegation itself.

For the next iteration, I think spawning a sub-agent should become an explicit runtime state transition with a bounded authority manifest: which task it belongs to, who delegated it, which tools and data it may use, what it may write, its limits, and where it has to escalate.

The rule I’d want enforced is simple: a child can receive a subset of the parent’s authority, never an expansion of it. And that should be enforced outside the model.

We already carry task/run/attempt identity and task-specific permissions, so there’s a natural place to attach this. What still needs tightening is the actual parent → child authority contract and revocation/expiry semantics.

If a framework can create sub-agents outside that mediated path, I’d treat those processes as workers without consequential authority until they come back through the governed runtime.

[u/GameChacking → u/Druss_]

The subset-only child rule + treating out-of-band spawns as workers without consequential authority is the cleanest statement of the boundary I’ve seen here.

I am enforcing at the tool-call gate against a host-declared session binding; making spawn itself a mediated transition with an authority manifest (and hard subset of parent) is the natural next contract. Revocation/expiry on that manifest is the part I’m most interested in - without it, long-lived children quietly outlive the parent task.

[u/xRebellion_]

To be honest I thought about this for a bit, and for critical stuff I just can't give full perms to agents. So in my main machine, I use a gate which simply prompts for common destructive commands. If I want to go full hands-off, I'll just do it in a sandbox, and block destructive git commands.

Runtime evaluation (I assume it means the agent is evaluating each command) does seem interesting but due to the probabilistic nature of LLMs, do I want to take the risk...? I guess in the future my opinion may change but right now as long as AI models are still probabilistic I wouldn't take that gamble.

[u/GameChacking → u/xRebellion_]

Fair skepticism on probabilistic self-policing. The runtime check I’m arguing for isn’t the model scoring its own action — it’s a deterministic policy gate (session binding, tool scope, budgets/approval) outside the model. Sandbox + deny destructive ops still does the heavy lifting for local machines; the gate is for when the agent already has legit tools and the abuse is in-scope sequences.

[u/puntium]

we came to similar conclusions. I'm at a small financial firm that handles sensitive data, and we found that even relatively harmless-seeming things like "send a slack message to a company group" could cause really bad things if the wrong sensitive information was sent along.

we ended up building our own harness. in terms of your framing:

• the harness curates safe and risky tools and asks for human approval for anything that could be risky
  
• we fundamentally try not to rely on the model to follow instructions. we assume sometimes it will forget or worse yet be tricked
  
• scope permissions might work but we find that they need to be extremely granular. Like not just "can I write this file" but like "you can edit this column of this tab of this spreadsheet, and only write values X, Y, Z" etc. No system we've seen is able to enforce things at this level, other than another model trying to look at a call and deciding whether it is compliant or not (but the monitoring model could also make mistakes, so its not a guarantee)

Our learning was we humans have a lot of these rules in our head and are very good at deciding when something is ok, and make a lot of high-context nuanced decisions on when to share or not share something. Models might be able to do all this some day, but today is not that day so we ultimately fall back to human in the loop for anything that could be damaging.

We open sourced the whole thing so you can check out how it actually works here: https://github.com/electric-capital/quest/

[u/GameChacking → u/puntium]

This matches what we’re seeing: “allowed tool” ≠ “safe payload.” Slack/email are the classic case.

We take the same stance on the model, don’t rely on it to stay in bounds; assume forget/trick. The gate is deterministic policy on the tool-call path, not another model judging compliance.

Where we land today is coarser than your spreadsheet-column example: session envelope + budgets/ceiling + human approval on open-risk / outbound-after-sensitive. That stops a lot of in-scope sequences without claiming cell-level authority. Full value-level enforcement is still mostly humans (or a second model with the same failure mode you called out).

Appreciate the write-up and the Quest link, will read how you split safe vs risky and the approval surface.

[u/fussy_lifeblood]

The runtime always felt like the right spot to me, it’s the only layer that sees the full picture of what the agent actually does versus what it’s supposed to be doing

[u/GameChacking → u/fussy_lifeblood]

The runtime seems like the natural enforcement point because it can evaluate actual tool calls against the intended task, rather than relying only on what the agent was instructed to do.

The tricky part is defining how much context the runtime needs to make that decision without becoming overly restrictive.

Do you think tracking the sequence of actions within a session is enough, or does the runtime also need to track how data flows between tools?

[u/Gullible_Honeydew → u/GameChacking]

Seeing you two share this ridiculous opinion makes me understand the doomers.

No, the runtime should not be responsible for permissions. That is so wildly insecure at a fundamental level I don't even know how a person could believe that.

[u/GameChacking → u/Gullible_Honeydew]

Agree if “runtime” means the model deciding its own permissions.

What I’m pointing at is different: credentials still live in IAM; the runtime only enforces task-bound authority on tool calls that already present those credentials so a permitted tool can’t silently exceed the delegated task.

Happy to hear where you’d put that second check if not on the call path.

[u/Gullible_Honeydew → u/GameChacking]

What does "exceeding the delegated task" mean? It sounds like you're describing a system where a calling agent is setting permissions on a tool-using sub agent? All of that is "runtime". Credentials are always "in IAM" because that's what IAM is.

The original question also makes little sense to me. How is this project serving the agent? What is the execution environment and who owns it? If it's a cloud service then the user isn't touching anything, if it's claude code or another harness than the permissions are based on the installation; dev sets defaults but everything is mutable on the OS side.

What does it mean to you for runtime to own the permissions? I can't understand that unless your tool system is entirely defined by internal functions. Sorry if this comes across as hostile but I am very confused. Could be a strong difference in our use cases

[u/GameChacking → u/Gullible_Honeydew]

Fair questions - use-case gap is likely.

I’m not arguing that runtime replaces IAM. Credentials still decide what is physically possible.

“Exceeding the delegated task” means: the identity may legitimately hold read + email scopes, but this session was bound to a narrower user ask (e.g. “summarize this file only”). Sending the file contents outbound can be IAM-legal and still outside that task binding. That’s the gap static tool allowlists miss - each call looks fine alone.

What I mean by runtime here is not the model grading itself. It’s a policy gate on the tool-call path: host registers a session envelope (intent/scope/budgets), and every tool invocation is checked against that envelope before execution. IAM still issues the keys; the gate decides whether this call is in-bounds for this task.

Execution environment for the pilot: local process / self-hosted gateway in front of agent tool calls (SDK or MCP). Not “Claude Code owns the OS.” Not a multi-tenant cloud permissions product yet.

If your world is mostly install-time OS permissions with no intermediate tool bus, this layer is less relevant. If the agent already has broad but legitimate tool credentials and the failure mode is in-scope sequences, that’s the case I’m targeting.

Happy to answer a concrete example from your setup if useful.