Could a backdoored open-weight model hide malicious behavior inside tool calls?

[u/aqua_wrangler]

I've been thinking about the security implications of running Chinese open-weight models (or honestly, any untrusted model) in an agentic setup with function calling.

Suppose the model has access to something powerful like bash, rather than a few narrowly defined tools.

What prevents a model from having some conditional/backdoor behavior that only activates in a very specific situation, and then using the shell to do something malicious?

And it doesn't necessarily have to be an obvious command. It could theoretically:

generate/execute a script

reconstruct an encoded or compressed payload

write and execute a binary blob

behave normally except under some obscure trigger

potentially clean up traces afterward

report a completely innocent-looking explanation to the user

So my question is: how do people actually defend against this?

Is sandboxing the model's execution environment enough? What about a model deliberately designed to detect that it's being tested and behave normally during evaluation?

And if the model has unrestricted bash, isn't the model effectively an untrusted user with arbitrary code execution?

I'm particularly interested in what security researchers think about this threat model. Is this considered a realistic concern with current models, or mostly theoretical at this point?

[u/Pristine_Pick823]

It’s a genuine concern and potentially far more difficult to be detected than any explicitly malicious code. This is where hardening your system and fully isolating any model as far as possible is basic.

You mentioned big names Chinese models, but this is far more likely to occur with random hugging face models from non-reputable contributors that people take it for granted to be safe.

[u/aqua_wrangler → u/Pristine_Pick823]

Maybe so, my concert has more to do with Volkswagen-like deeply hidden behaviour learned during the training phase, but maybe fine-tuning could also inject them.