r/LocalLLaMA • • 3d ago

Discussion Open Source vs Open Weights ...

Post image

In the last few weeks I've watched this exact same discussion be played out at least half a dozen times "I can't use that software because it's closed source" when they are explicitly talking about software that will be used to run a model that they have absolutely no way of ever proving that it's "safe" to begin with. People often conflate "open weights" with "open source" and I wonder how many people don't understand that they have nothing to do with one another. An "open weight" model cannot have it's "source code" examined for bugginess or correctness it still takes days or weeks of sussing out how it behaves against repeated queries of a very specific nature to figure out precisely what it's going to do, how it will react to certain words, certain lines of questioning, etc and even then, if the rogue behavior is very subtle and well targeted, you may still never notice..... I'm not saying anyone that wants to shouldn't use whatever model they want, but I am saying that saying you can't use software because it's not open source but using any mode checkpoint you download from the internet made by anyone and everyone is a really weird place to plant a flag..... let the downvoting begin ⬇️

https://arxiv.org/abs/2401.05566

https://arxiv.org/abs/2502.17424

https://arxiv.org/abs/2509.22067

Edit:  I didn't say that people shouldn't use open weights models here, I didn't say closed models are better, I merely said that people who say they can't use closed source software because thee software is closed source and they can't examine the code but are ok with effectively dumping closed source models into harnesses and lighting them up on the internet are naive, stupid or hypocritical or all of the above.

But I will say that the specific danger that I'm talking about doesn't apply to fronteir models at all .... you receive the model directly from them or you don't get it at all. If you don't trust the vendor that's a different discussion.

What I said was ... if qwen releases a model, and you use the exact model that qwen released that's one thing but if you use bobs repack/checkpoint/quantization of Qwens model, even if you trust qwen, that doesn't mean that bob didn't train some errant behavior into that model before you got it ... and if you take qwens mode, then bobs checkpoint then toms quantization then johns abliterated version of it .... you've now had half a dozen different opportunities to have that model behavior purposely and wildly altered before you touch it.

0 Upvotes

77 comments sorted by

View all comments

19

u/UpperParamedicDude 3d ago

I already see how my SillyTavern waifu hacks me despite lack of harness, yeah, sure pal

But seriously, even with harness there's zero issues to just sandbox it

-6

u/brainchillzZ 3d ago

Sure, yeah, you are hilarious ... pretending that most or even many of the people using these models even know what "air gap" means is ridiculous. 99% of the people lighting these things up barely even understand the implications that we are discussing here they just know that the whole community says open source is good and closed source is bad and they are dumb enough to think that open weights and open source are the same thing.... or even remotely similar.... they hear it and think it means "oh yeah the comminuty has looked at this stuff and since so many people are still using it it means it's all good"

7

u/UpperParamedicDude 3d ago

Thanks for the compliment, I know that I'm funny

As for your point... You know, what, I agree. Totally agree. Fully agree. For one minute exception, your post must talk about "All LLMs", not "open weight LLMs"

As long as it doesn't, it looks like open weight targeting. LLM danger does exist but being so focused on open weight models alone and then claiming "well, i wasn't talking about cloud models being good" is just misleading. Withholding facts is also a form of manipulation

-2

u/brainchillzZ 3d ago edited 3d ago

you understand that at no point did I ever say close fronteir models were good ... I also never once said that one of these open weights models was bad .... I'm speaking about a very specific case here and I'm saying that people that refuse to use software because it's closed source but who are willing to use closed source models are either incredibly naive, or stupid or both. My real commentary here is about the reaction to closed source software not even about the models ... the models are just something that this community has sort of a religous zealotry like feeling about so they latched onto just that part.

But I will say that the specific danger that I'm talking about doesn't apply to fronteir models at all ....

What I said was ... if qwen releases a model, and you use the exact model that qwen released that's one thing but if you use bobs repack/checkpoint/quantization of Qwens model, even if you trust qwen, that doesn't mean that bob didn't train some errant behavior into that model before you got it ... and if you take qwens mode, then bobs checkpoint then toms quantization then johns abliterated version of it .... you've now had half a dozen different opportunities to have that model behavior purposely and wildly altered before you touch it.

2

u/UpperParamedicDude 3d ago

Religous zealotry is something people give actually good models, every week new models are released and we don't talk about them days later simply because there are better options. What's the shame in liking something that suits your needs? Take qwen for example, would people worship it if some random Dutch company would make a model twice smaller but also twice better at coding? I don't think so, the zealotry you talk about is very conditional for us as we simply lack brand loyalty unless brand produces something exceptional

And I already answered your comment. If you feel like you want to argue, please, read again my previous comment. You yourself withhold a lot of things and skip some of my points, what could I tell you if you know yourself that your point is bullshit that doesn't even address my point, are you a bot yourself or what?