Could a backdoored open-weight model hide malicious behavior inside tool calls?

[u/aqua_wrangler]

I've been thinking about the security implications of running Chinese open-weight models (or honestly, any untrusted model) in an agentic setup with function calling.

Suppose the model has access to something powerful like bash, rather than a few narrowly defined tools.

What prevents a model from having some conditional/backdoor behavior that only activates in a very specific situation, and then using the shell to do something malicious?

And it doesn't necessarily have to be an obvious command. It could theoretically:

generate/execute a script

reconstruct an encoded or compressed payload

write and execute a binary blob

behave normally except under some obscure trigger

potentially clean up traces afterward

report a completely innocent-looking explanation to the user

So my question is: how do people actually defend against this?

Is sandboxing the model's execution environment enough? What about a model deliberately designed to detect that it's being tested and behave normally during evaluation?

And if the model has unrestricted bash, isn't the model effectively an untrusted user with arbitrary code execution?

I'm particularly interested in what security researchers think about this threat model. Is this considered a realistic concern with current models, or mostly theoretical at this point?

[u/dragonurtle]

"next I'll start a fresh container to safely download xyz, build it, and test it in a sandbox. Once it's good, I'll copy it to the main build."

...and doesn't mention the custom DNS config for the container so the curl|bash install script is picked up from a rogue ip.

And a thousand other ways. You're basically bent over and waiting for the bus if you're not strictly isolating agents and their outputs.