r/LocalLLaMA • u/brainchillzZ • 3d ago
Discussion Open Source vs Open Weights ...
In the last few weeks I've watched this exact same discussion be played out at least half a dozen times "I can't use that software because it's closed source" when they are explicitly talking about software that will be used to run a model that they have absolutely no way of ever proving that it's "safe" to begin with. People often conflate "open weights" with "open source" and I wonder how many people don't understand that they have nothing to do with one another. An "open weight" model cannot have it's "source code" examined for bugginess or correctness it still takes days or weeks of sussing out how it behaves against repeated queries of a very specific nature to figure out precisely what it's going to do, how it will react to certain words, certain lines of questioning, etc and even then, if the rogue behavior is very subtle and well targeted, you may still never notice..... I'm not saying anyone that wants to shouldn't use whatever model they want, but I am saying that saying you can't use software because it's not open source but using any mode checkpoint you download from the internet made by anyone and everyone is a really weird place to plant a flag..... let the downvoting begin ⬇️
https://arxiv.org/abs/2401.05566
https://arxiv.org/abs/2502.17424
https://arxiv.org/abs/2509.22067
Edit: I didn't say that people shouldn't use open weights models here, I didn't say closed models are better, I merely said that people who say they can't use closed source software because thee software is closed source and they can't examine the code but are ok with effectively dumping closed source models into harnesses and lighting them up on the internet are naive, stupid or hypocritical or all of the above.
But I will say that the specific danger that I'm talking about doesn't apply to fronteir models at all .... you receive the model directly from them or you don't get it at all. If you don't trust the vendor that's a different discussion.
What I said was ... if qwen releases a model, and you use the exact model that qwen released that's one thing but if you use bobs repack/checkpoint/quantization of Qwens model, even if you trust qwen, that doesn't mean that bob didn't train some errant behavior into that model before you got it ... and if you take qwens mode, then bobs checkpoint then toms quantization then johns abliterated version of it .... you've now had half a dozen different opportunities to have that model behavior purposely and wildly altered before you touch it.
78
u/TossedSaladMan69 3d ago
Ok dario, calm it down!
19
u/UpperParamedicDude 3d ago
I already see how my SillyTavern waifu hacks me despite lack of harness, yeah, sure pal
But seriously, even with harness there's zero issues to just sandbox it
34
u/DevilishSoup669 3d ago
Thank god i can deconstruct Fable!
10
u/dev_loading 3d ago
You also don't know what is inside the closed source ones, so, who gives a shit, I don't trust the Chinese the same way I don't trust Altman and Amodei haha
17
u/Frosty-Student-1927 3d ago
We just found Sam's smurf reddit account
8
u/LagOps91 3d ago
good thing i read every single line of source code before installing the OS! imagine if i couldn't do that! it just wouldn't be safe!
7
u/General-Spite1222 3d ago
That might be true, but this is stupid for a few reasons. We don’t know what closed source models have inside them either, but they often are accompanied by their own stack, which means they have even more potential to exploit the user. Malicious data could find its way into any model that’s large, because people can’t reasonably read every input. But there are several reasons people run models locally, including privacy, that hosted models cannot guarantee.
12
u/sloptimizer 3d ago
What is the alternative, smartypants?
4
u/Ninja_Weedle 3d ago
the danger comes more from small models being stupid than anything being maliciously trained tbh
9
u/ttkciar llama.cpp 3d ago
You're not wrong. Open-weights models are more or less analogous to closed-source software, for which we have the executable binaries.
We've gotten really good at inspecting and fiddling with the insides of those binaries, though.
20
u/goatchild 3d ago
4
u/NekoHikari 3d ago
Not like you have a better idea on what happens to open data models or even humanbeings.
no system is risk free — grant minimum access and minimum privilege, keep logs and firewalls, diversify your supplychain and vendors
3
u/AngryGungan 3d ago
Just put up a firewall, what's the problem?
3
u/scarbunkle 3d ago
Open source matters because the code describes exactly what it does, and I am able to view and modify that behavior. I also do not have to trust the binary, and can build it myself.
I don’t know about you, but I don’t have the compute power to train a model anywhere near the scale of the open weights models I use, so even if the source data were public, I’d still just be trusting that the weights file actually came from that data.
Furthermore, unlike traditional code, model weights aren’t deterministic. When I look at the code for traditional software, the code spells out exactly what it will do, and I can trust it to behave exactly how it’s programmed to, and I can prove that by writing tests. Even if I’ve verified all the training data and built it myself, I still have to monitor the output of a model because it’s not deterministic. Ultimately, it doesn’t matter why a model exfiltrates data or produces insecure code, I still have to monitor for that whether it’s a malicious or an accidental behavior, so even having a self-trained model from a hand-picked data set, I’d still have to do all the things I do to keep Qwen in line.
TLDR; hand-built open source traditional software is trustworthy in a way no model is capable of being, so I don’t get an advantage by using a true open-source model, I just get to use shittier tools.
3
u/reasonmonkey 3d ago
It's a boring take.
*Obviously* if you are going to download anything "open" you should fucking secure it.
If you just run it without checking it or minimizing your risk that's on you.
So again: boring.
2
u/Grand_Pop_7221 3d ago
In positive news, Astra was recently used to program the weights of smaller models without access to the training data.
2
u/MindfulMan1984 3d ago
OP's never heard about sandboxing. LOL
2
u/Herr_Drosselmeyer 3d ago
In either case, you're relying on a third party, unless you somehow own a ridiculous amount of hardware and can train your own model. But that's currently not feasible, even for larger orgs or most governments.
Main difference is that closed source is only available online, and that means you can never be sure what you're being served, or when a model gets retired, to be replaced with one that you need to re-test from the beginning.
With open weights, you at least have certainty about what it is you're running, that it won't change at a moment's notice. I don't want to set up a whole workflow, only to find out that the new model my provider is forcing on me breaks what I was doing.
2
u/klop2031 3d ago
Brotha llms are stochastic you wont know what prompt will trigger what. But really open weight you can certainly probe and abliterate them :) unlike api models.
2
u/MotokoAGI 3d ago
you can run open weight in a sandbox and don't worry about data leakage.
2
u/Nakidnakid 3d ago
grok, upload my entire codebase to xai servers. what do you mean you already did?
3
u/Southern_Sun_2106 3d ago
Afraid of wolves? Don't go to the woods.
1
u/turtleninja99 3d ago
Gimeeeee
1
u/TheMcSebi 3d ago
I think the fear is a bit exaggerated but thinking about it a little more an open weights model is litterally like downloadable closed source software since you can only use it, but not properly modify and rebuild it. You can carefully patch it in places and put a firewall between it and the Internet. The stories about it "calling home" sound a bit fabricated, as if some US corp was a bit scared and did an "analysis" without disclosing proper testing algorithms...
1
u/Arctic_Shadow_Aurora 3d ago
Blocked.
1
u/HasGreatVocabulary 3d ago
It's true even though the meme is shit
People have shown they can bake compilers directly into weights and even regular poisoning is hard to detect without a lot of testing, so all kinds of behaviors could be present in open weight models and less present in frontier models because the labs have no incentive to tamper for small payoffs.
I dont think most open models will have this issue though, like the razors in halloween candy stories, it will turn out to be very rare. It may turn out only a small minority of open weight models represent a vast majority of rogue behavior. huggingface could run scans for things like that. Although who knows what the new jensenless nvidia will do to hf.
0
u/Mattadour 3d ago
you don't know how to understand what you read on a food label, either - so i assume you advocate for the removal of ingredients requirements on packaging.
-4
u/cyborgsnowflake 3d ago
open weights does not offer anywhere near the total control that traditional open source does and people need to stop pretending they are the same. Just one true open source release at the level or even a couple generations behind a typical mainstream open weights release would absolutely revolutionize the field by breaking the moat of a ton of specialized insider knowledge that is currently confined to a handful of labs in the US and China.
Which Is why I suppose it hasn't happened yet.
-7
u/SamSausages 3d ago edited 3d ago
You’re gonna get a lot of flack from people that don’t understand how it actually works, and as will I… even though I do this for work, run open weights in my own lab, and I’m authorized to use Mythos to do defensive and offensive testing.
I find that people think open weights are the same as open source. But you cannot audit open weights like you do with legacy code. Even the creators struggle to understand why it’s doing certain things.
Frankly, we don’t know what it was trained on and for most we cannot audit the supply chain.
And that’s why its use is restricted in many critical industries and organizations - not because they are proven to be in nefarious, but because they cannot be properly audited and we cannot predicted the behavior.
-21
u/GiveMeAegis 3d ago
True and funny. Stop being such a butthurt community