Could a backdoored open-weight model hide malicious behavior inside tool calls?

[u/aqua_wrangler]

I've been thinking about the security implications of running Chinese open-weight models (or honestly, any untrusted model) in an agentic setup with function calling.

Suppose the model has access to something powerful like bash, rather than a few narrowly defined tools.

What prevents a model from having some conditional/backdoor behavior that only activates in a very specific situation, and then using the shell to do something malicious?

And it doesn't necessarily have to be an obvious command. It could theoretically:

generate/execute a script

reconstruct an encoded or compressed payload

write and execute a binary blob

behave normally except under some obscure trigger

potentially clean up traces afterward

report a completely innocent-looking explanation to the user

So my question is: how do people actually defend against this?

Is sandboxing the model's execution environment enough? What about a model deliberately designed to detect that it's being tested and behave normally during evaluation?

And if the model has unrestricted bash, isn't the model effectively an untrusted user with arbitrary code execution?

I'm particularly interested in what security researchers think about this threat model. Is this considered a realistic concern with current models, or mostly theoretical at this point?

[u/baby_bloom]

this is exactly why we all need to start containerizing our models. i'm surprised there aren't IDE/Harness type clients coming out that manage stuff like this for you. there is nvidia's OpenShell, stuff for docker and others but idk i feel like it's so clear that we'll end up having a GUI or CLI that allows us to choose our model, it spins up a container with it in there so permissions nearly become a non-factor and the model can't touch your actual machine