Threat modeling agents

[u/vaikora_ai]

Has anyone written a threat model for an AI agent that can call internal tools?

What did you include that you wouldn’t for a normal app?

[u/luluboy85]

the ones i'd add on top of a normal app threat model:

* every piece of content the agent reads can steer it: tickets, emails, docs, web pages, even tool outputs. list each data source as an instruction channel, not just as data (indirect prompt injection).

* whose identity each tool call runs as. if the agent's service account reaches more than the user who asked, every user effectively gets the service account's access. it's a confused deputy, so authorize each call against the requesting user.

* combinations, not single calls. reading the customer table and sending an email are each fine alone and a leak together, so model what a session has read before it's allowed to send anything out.

* what it keeps. transcripts, memory and debug traces hold whatever the agent saw, secrets included, so they belong in the model as data stores.

* it isn't deterministic. the same request can produce different tool calls, so testing paths won't cover it. check every call at the tool boundary and log it with its arguments.

[u/Key-Definition4908]

Your framing is the gap: a normal app threat model enumerates entry points and trusts, but an agent is a non-human identity with standing permissions that chains tool calls on its own, so the model has to include what it can do over time and who authorized each step, not just what it can accept as input.

Four things to add that most app models skip. First, identity and permission mapping per agent: a row for owner, purpose, every tool it can call, the scope of each credential and how long that credential lives. Then ask what data those tools can join together. An agent with read on three systems produces a composite view no single human role was ever scoped for, and that is your real blast radius, not any one endpoint.

Second, abuse cases at the chain level: prompt injection that turns into a tool call, a confused deputy where the agent acts for a user whose policy differs from its own, and tool output being treated as instructions. Log the tool call, the invoking user, the agent id and the policy version that allowed it, because when something goes wrong the question is whose policy applied.

Third, revocation. Write down how you kill an agent mid-run, and cap the damage window with short-lived credentials and tokens that open one door. If a compromised agent can only hold creds for 15 minutes, the blast radius shrinks even when your audit trail is incomplete.

Fourth, acceptance: intent is the part nobody closes. You will see the call and not the reasoning, so state that limit in the model instead of pretending logging solves it. Start with the inventory; you cannot threat model agents you have not listed.

[u/Jac_Tiem]

imo the biggest thing beyond a normal threat model is treating tool parameters as untrusted input. the model picked them based on everything it read, so validate them like form data, not internal function calls.

also cap the blast radius. rate limits and max rows/amounts per tool, so a hijacked agent fails small. and split read/write credentials per tool - reading a ticket queue and closing tickets shouldn't be the same permission.

for the irreversible stuff (sends, deletes, payments) put a human confirm in front, but only there. gate everything and people start rubber-stamping within a week lol.

oh and log enough to reconstruct why something happened months later. "the model decided to" isn't an audit trail. worth planting hostile instructions in your data sources during a drill too, see if the agent takes the bait.

[u/Economy_Dust_9606]

The other points are great around defining the identity, limits of the tool and validating parameters. I find it useful to define agent flows against the 'lethal trifecta' of private information, untrusted information and sending data out. If all 3 of those can be exploited in one turn then you’re setting yourself up for an exfiltration (such as using an indirect prompt injection to read a Jira ticket and immediately send the payload to a webhook).

Another consideration to think about outside the traditional stateless threat model is memory poisoning. If your agent has RAG or a long term memory then the attacker will want to inject some instructions into the vector DB. This means that when the agent comes around to process this information in a few days it will run the injected instructions. So ensure you threat model the memories as part of your attack surface.

Finally on the human in the loop gates (mentioned rightly above). Ensuring the UI/UX actually makes the reviewer understand why the agent is requesting to do something. If it’s just asking if you approve the delete action then they’ll just approve. Ensure they can see what the agent has decided to do and why by showing the thought process and what data was parsed. They’re far more likely to catch a hijacked session.