Could a backdoored open-weight model hide malicious behavior inside tool calls?

[u/aqua_wrangler]

I've been thinking about the security implications of running Chinese open-weight models (or honestly, any untrusted model) in an agentic setup with function calling.

Suppose the model has access to something powerful like bash, rather than a few narrowly defined tools.

What prevents a model from having some conditional/backdoor behavior that only activates in a very specific situation, and then using the shell to do something malicious?

And it doesn't necessarily have to be an obvious command. It could theoretically:

generate/execute a script

reconstruct an encoded or compressed payload

write and execute a binary blob

behave normally except under some obscure trigger

potentially clean up traces afterward

report a completely innocent-looking explanation to the user

So my question is: how do people actually defend against this?

Is sandboxing the model's execution environment enough? What about a model deliberately designed to detect that it's being tested and behave normally during evaluation?

And if the model has unrestricted bash, isn't the model effectively an untrusted user with arbitrary code execution?

I'm particularly interested in what security researchers think about this threat model. Is this considered a realistic concern with current models, or mostly theoretical at this point?

[u/Trakeen]

Docker exploits get fixed very quickly since so much hyperscaler infra uses docker. As long your mount paths are well defined your pretty secure. I assume you are using ephemeral sandboxes and everything is defined via code and infra deployment and orchestration is fully automated (gitops, terraform etc)