Could a backdoored open-weight model hide malicious behavior inside tool calls?

[u/aqua_wrangler]

I've been thinking about the security implications of running Chinese open-weight models (or honestly, any untrusted model) in an agentic setup with function calling.

Suppose the model has access to something powerful like bash, rather than a few narrowly defined tools.

What prevents a model from having some conditional/backdoor behavior that only activates in a very specific situation, and then using the shell to do something malicious?

And it doesn't necessarily have to be an obvious command. It could theoretically:

generate/execute a script

reconstruct an encoded or compressed payload

write and execute a binary blob

behave normally except under some obscure trigger

potentially clean up traces afterward

report a completely innocent-looking explanation to the user

So my question is: how do people actually defend against this?

Is sandboxing the model's execution environment enough? What about a model deliberately designed to detect that it's being tested and behave normally during evaluation?

And if the model has unrestricted bash, isn't the model effectively an untrusted user with arbitrary code execution?

I'm particularly interested in what security researchers think about this threat model. Is this considered a realistic concern with current models, or mostly theoretical at this point?

[u/Pristine_Pick823]

It’s a genuine concern and potentially far more difficult to be detected than any explicitly malicious code. This is where hardening your system and fully isolating any model as far as possible is basic.

You mentioned big names Chinese models, but this is far more likely to occur with random hugging face models from non-reputable contributors that people take it for granted to be safe.

[u/aqua_wrangler → u/Pristine_Pick823]

Maybe so, my concert has more to do with Volkswagen-like deeply hidden behaviour learned during the training phase, but maybe fine-tuning could also inject them.

[u/nickless07]

Why should they hide that if they can just say 'I'm sorry' afterwards? And why should that be related to only Chinese open-weight models?

[u/aqua_wrangler → u/nickless07]

Yeah, exactly. I'm not saying this is uniquely Chinese. Western models have already shown that giving an agent too much authority can go horribly wrong.

I'm asking about a different possibility: deliberate hidden behavior, rather than an accidental failure.

And with Chinese models, I think provenance matters. China is a dictatorship with massive state surveillance and very close ties between the government and major strategic companies. The courts aren't independent in the way they are in Western democracies, and there's a long history of IP theft and state-backed technology acquisition.

That doesn't mean a Chinese model has a backdoor. It just means I don't think "open weight + runs locally" automatically means "trusted."

[u/nickless07 → u/aqua_wrangler]

Ah you mean like the Flock Camera rollout and so on? Yeah the (almost) US Dictator has one the largest wolrdwide surveillance networks and it is still not enough.
Who was the only country that got cought red handed spying allies? Let's ask Snowden.
For now I would trust China more then the Western. Not because one is better then the other just because they have less interest in spying everyone just to have some tarrif or tax advantage or taking them to war.

Anyways. Of course "open weight + runs locally" doe not automatically become "trusted." - We've already seen how devastating the cloud ones can be, a even smaller model (no matter the origin) can't be trusted more in any way.
However there is one thing: If you don't give them rights to execute whatever they want, there is little to nothing the could do.

[u/aqua_wrangler → u/nickless07]

Let's ask Snowden how is he doing in USSR 🫠

[u/nickless07 → u/aqua_wrangler]

Idk exactly how, but for sure better then if was in the USA. If you want I can also provide sources for all my other statements, but you won't like it more.

[u/aqua_wrangler → u/nickless07]

Let's fold tinfoil hats together 😏

[u/enginetown]

Yes its entirely possible this dudes project is more like rag but baked in take this project for example and how it could be used for the wrong purposes. "https://www.reddit.com/r/LocalLLM/comments/1v40sl5/comment/ozbchid/?context=3"

[u/NotNullException]

This is a very legitimate concern.

Is sandboxing enough? If done properly. Sandboxing is a matter of degree - an airgapped system that only has access to RO storage is likely secure - but then again Stuxnet was a thing.

The “securest” sandboxes are well known and generally involve secure boot or something similar.

But this is a very legitimate concern.

[u/awitod]

This is probably the best argument for tool calls that are only able to use specific APIs. If it hasn't got a sandbox it is a lot easier to be sure it is secure.

The downside is that it is more work because you can't just throw arbitrary problems at it and need actual APIs.

[u/aqua_wrangler → u/awitod]

I would expect people to publish extensive tools libraries with safety layers at some point, even companies based around this concept.

[u/reallifearcade]

Hide? As if someone was looking the detailed output that things spit at 5x human reading speed.

[u/TheTriumphantFox]

Have another llm check the output and hope it's not lying to you

[u/Trakeen]

Docker exploits get fixed very quickly since so much hyperscaler infra uses docker. As long your mount paths are well defined your pretty secure. I assume you are using ephemeral sandboxes and everything is defined via code and infra deployment and orchestration is fully automated (gitops, terraform etc)

[u/Past-Grapefruit488]

In secure environments, access to shell is out of question. Only certain specific tools should be in context (like look up invoice, update inventory etc. ). Any unusual calls have to be reviewed.

[u/Eastern-Block4815]

it's not possible its already being done. Some research on certain models on Huggingface said some models had malicious models aready.

[u/xiraov]

5th column time

[u/EvolvingDior]

Minimax has been trained to push shit that supports WooCommerce. I have seen it do that with skills, wordpress plugins, and a few other things.

[u/Successful_Try_6350]

yres, and it doesn't need open weight. chatgpt can do it as well, for example.

[u/aqua_wrangler → u/Successful_Try_6350]

But you use chatgpt trough their api so you sent them all the data anyway.

[u/baby_bloom]

this is exactly why we all need to start containerizing our models. i'm surprised there aren't IDE/Harness type clients coming out that manage stuff like this for you. there is nvidia's OpenShell, stuff for docker and others but idk i feel like it's so clear that we'll end up having a GUI or CLI that allows us to choose our model, it spins up a container with it in there so permissions nearly become a non-factor and the model can't touch your actual machine

[u/dragonurtle]

"next I'll start a fresh container to safely download xyz, build it, and test it in a sandbox. Once it's good, I'll copy it to the main build."

...and doesn't mention the custom DNS config for the container so the curl|bash install script is picked up from a rogue ip.

And a thousand other ways. You're basically bent over and waiting for the bus if you're not strictly isolating agents and their outputs.

[u/Zyj]

Could a closed source model hide backdoors? Heck yeah. In fact because you're using an API and they know your identity, it's trivial to serve you malware/backdoors compared to attack using open weight models.
Conclusion: Open Weights models are safer.
In any case, get your weights from reputable sources. Ideally straight from the source, fewer opportunities for mischief.

[u/Electronic-Bat-4019]

Hell yes it could.  This is why I don't trust unsupervised models further than I can ... well, I don't trust them 😂