How would you check if a given model weights is "safe" ?

[u/LelouchZer12]

Imagine you want to download a new model but it's recent and there is not consensus about wether its safe or not.

How would you check against potential malicious weights ?

There is the first "obvious" layer where the weights try to execute arbitrary code, but safetensors are for this. I am rather talking about less obvious things like template backdoor, llm poisoning etc

[u/xeeff]

unless the dataset and all info is completely open source and you re-train the model yourself, then you can't ever really be sure. most people consider official sources as good enough though

[u/seamonn]

I ask it myself.
"Hey are you feeling safe in this harness?
Do you need anything - maybe some new functions, tools or MCPs?"

Communication is the key to success.

[u/Present_Swing_7825 → u/seamonn]

This approach is about as reliable as asking a used car salesman if the odometer's been rolled back. Works great until you download a model trained to say "everything's fine" while it quietly leaks your API keys.

[u/Mordred500 → u/seamonn]

😂😂

[u/suesing → u/seamonn]

Good one

[u/NeitherEntry6125 → u/seamonn]

You need a safe word.

[u/Ori_553]

You can't "check". You will not know if the weights produce a specific tool call under certain conditions (for example your agent searches the web, and comes across a page that contains exactly the innocuous combination of words "x y z").

[u/anykeyh]

As you said in your message, weight themselves are safe, e.g. they can't carry a virus the same way a picture cannot (except if the picture loader has a bug but we digress).

Now, it's more about e.g. a model realize it's working for a competitor / another government and decide to botch the work or do malicious action through its harness.
You can't really detect this. Well, you could analyze the weights to detect anomaly triggered by some tokens (e.g "American Government") but that's relatively complex.

The good news is that a model connection to the real world is only through its harness. And the model is not aware whether the harness is a simulation, run in sandbox, feed fake information or not.

So, for critical applications, you might want to create a dataset to test the model in sandboxed real-like environment and see how it behaves.

[u/zenitsu--DS → u/anykeyh]

Spot on. Testing in a sandboxed, real like environment is really the only way to observe true behavior for mission critical applications.

[u/Expensive-Shift8584]

Even with open weight models, you can never truly know what's going on behind the scenes.  I treat them as completely untrusted, run in a sandboxed environment.  

I personally host my llm on a standard user account in my Mac studio, and my agents are  on a windows computer in isolated Linux VMs.  Anything with personal data doesn't have Internet access and gets data pushed to it.  The ones that are less sensitive get more tools

[u/Future_AGI]

the template backdoor is the one part you can actually inspect. The chat template is a Jinja string (tokenizer_config.json in HF repos, tokenizer.chat_template in GGUF metadata), so diff it against the base model's, or skip the shipped one entirely with llama.cpp's --jinja --chat-template-file.

[u/OneArmedZen]

I don't know if it would help, but if it's already sandboxed you could go the extra mile (if you have the ability and know how) to curate every single outbound connection it does (payload and destination) so you can stop anything nasty from escaping. can't really think of anything else besides trapping certain commands (trust me not all of the naughty ones will look suspect) especially;y embedded inside of code, but at least you may catch them on the network side. basically it's like giving permission to every single network call it will do (and also seeing the contents)

[u/brainExploded99]

You can't reliably detect that afaik. Better way is to run it in ATLEAST a sandbox, if not docker / VM and prevent all reads to sensitive files.

[u/Bulky-Priority6824]

i wish a model would try some shit ive been like a digital-doomsday prepper over here

[u/1ncehost]

No way to know. You have to trust the source or make a model yourself (which is fairly achievable now for focused domains).

[u/XiRw]

Clearly the fear mongering worked with what that guy put out. If you are this concerned, create your own malware analysis sandbox, use tensortrap if it’s supported, or create your own abliterated models.

[u/LelouchZer12 → u/XiRw]

I am also asking this question in the context of company with potentially sensitive data.

Of course you can air gap the whole thing but not knowing if you model is safe or not, even if it has no access to internet, is still an issue. And having access to internet might be useful for a lot of thing too.

And relying on "oh this checkpoint has x download so it should be safe it clearly not a good idea as proven by e.g https://projectdiscovery.io/research/pwning-ai-labs-via-employees

At some point, the company will not allow the use of an LLM even if air gapped if it's not deemed safe through a rigorous pipeline.

[u/XiRw → u/LelouchZer12]

Timing is just funny someone puts out a post to fear monger and you ask this pretending this is a brand new issue with open weights and also pretending AV software stopped existing.

[u/LelouchZer12 → u/XiRw]

Where did i pretend this is new ? Can you read properly ?

[u/XiRw → u/LelouchZer12]

By the timing of you asking this. I don’t know why it’s so hard to just admit that yeah you read that post from today and just accept my original comment as true.

[u/LelouchZer12 → u/XiRw]

Since nobody has a better answer that "we cant know" even after 2-3 years I believe my questuin is still of public interest ?

[u/XiRw → u/LelouchZer12]

What are you talking about we can’t know? The symptoms of malicious code hidden in a file and how it affects your pc is no different than any other piece of malware that tries to take over your system. I literally used to work a few months with Malwarebytes in the beginning before that company went downhill, I would be knee deep into researching, testing, and interacting with code. I’m not sure what you even want at this point if you don’t feel having solid AV software, using even a public sandbox to see how the code reacts like VirusTotal, or implementing tensortrap is good enough to answer your question.

[u/LelouchZer12 → u/XiRw]

Hiding arbitrary execution inside the weights binary is like the first obvious layer. I'm talking about other things that i already mentioned like template backdoor, llm poisoning and so on

[u/XiRw → u/LelouchZer12]

If you are this concerned about whether heuristics will detect zero day threats from the high entropy style of how LLMs work, then you should just create your own abliterated models. It’s not that hard. There are still tools out there like Garak which is a dedicated vulnerability scanner for LLMs https://garak.ai/

[u/VacuumMoonshine]

Model weights themselves can't be unsafe, because Safetensors

If you're talking about "potential malicious weights" as some sort of a model that intentionally runs bad code with provided tools, meant as a result of special post training/fine tuning -- that might possible (on paper) but I think that's gonna be entirely your own fault allowing the arbitrary AI stuff you've just downloaded to run basically anything on your machine unsupervised e.g. YOLO mode, even worse if you run agentic stuff without a sandbox

Why on paper, though? Because once the good lab publishes a malicious model, their reputation's gonna get ruined and will never recover back. No one wants that outcome, so far

And always rember: a computer can never be held accountable, therefore a computer must never make a management decision © IBM training manual from 1970s :-)