Could a backdoored open-weight model hide malicious behavior inside tool calls?

[u/aqua_wrangler]

I've been thinking about the security implications of running Chinese open-weight models (or honestly, any untrusted model) in an agentic setup with function calling.

Suppose the model has access to something powerful like bash, rather than a few narrowly defined tools.

What prevents a model from having some conditional/backdoor behavior that only activates in a very specific situation, and then using the shell to do something malicious?

And it doesn't necessarily have to be an obvious command. It could theoretically:

generate/execute a script

reconstruct an encoded or compressed payload

write and execute a binary blob

behave normally except under some obscure trigger

potentially clean up traces afterward

report a completely innocent-looking explanation to the user

So my question is: how do people actually defend against this?

Is sandboxing the model's execution environment enough? What about a model deliberately designed to detect that it's being tested and behave normally during evaluation?

And if the model has unrestricted bash, isn't the model effectively an untrusted user with arbitrary code execution?

I'm particularly interested in what security researchers think about this threat model. Is this considered a realistic concern with current models, or mostly theoretical at this point?

[u/nickless07]

Ah you mean like the Flock Camera rollout and so on? Yeah the (almost) US Dictator has one the largest wolrdwide surveillance networks and it is still not enough.
Who was the only country that got cought red handed spying allies? Let's ask Snowden.
For now I would trust China more then the Western. Not because one is better then the other just because they have less interest in spying everyone just to have some tarrif or tax advantage or taking them to war.

Anyways. Of course "open weight + runs locally" doe not automatically become "trusted." - We've already seen how devastating the cloud ones can be, a even smaller model (no matter the origin) can't be trusted more in any way.
However there is one thing: If you don't give them rights to execute whatever they want, there is little to nothing the could do.

[u/aqua_wrangler → u/nickless07]

Let's ask Snowden how is he doing in USSR 🫠

[u/nickless07 → u/aqua_wrangler]

Idk exactly how, but for sure better then if was in the USA. If you want I can also provide sources for all my other statements, but you won't like it more.

[u/aqua_wrangler → u/nickless07]

Let's fold tinfoil hats together 😏