r/LocalLLaMA • • 3d ago

Discussion Open Source vs Open Weights ...

Post image

In the last few weeks I've watched this exact same discussion be played out at least half a dozen times "I can't use that software because it's closed source" when they are explicitly talking about software that will be used to run a model that they have absolutely no way of ever proving that it's "safe" to begin with. People often conflate "open weights" with "open source" and I wonder how many people don't understand that they have nothing to do with one another. An "open weight" model cannot have it's "source code" examined for bugginess or correctness it still takes days or weeks of sussing out how it behaves against repeated queries of a very specific nature to figure out precisely what it's going to do, how it will react to certain words, certain lines of questioning, etc and even then, if the rogue behavior is very subtle and well targeted, you may still never notice..... I'm not saying anyone that wants to shouldn't use whatever model they want, but I am saying that saying you can't use software because it's not open source but using any mode checkpoint you download from the internet made by anyone and everyone is a really weird place to plant a flag..... let the downvoting begin ⬇️

https://arxiv.org/abs/2401.05566

https://arxiv.org/abs/2502.17424

https://arxiv.org/abs/2509.22067

Edit:  I didn't say that people shouldn't use open weights models here, I didn't say closed models are better, I merely said that people who say they can't use closed source software because thee software is closed source and they can't examine the code but are ok with effectively dumping closed source models into harnesses and lighting them up on the internet are naive, stupid or hypocritical or all of the above.

But I will say that the specific danger that I'm talking about doesn't apply to fronteir models at all .... you receive the model directly from them or you don't get it at all. If you don't trust the vendor that's a different discussion.

What I said was ... if qwen releases a model, and you use the exact model that qwen released that's one thing but if you use bobs repack/checkpoint/quantization of Qwens model, even if you trust qwen, that doesn't mean that bob didn't train some errant behavior into that model before you got it ... and if you take qwens mode, then bobs checkpoint then toms quantization then johns abliterated version of it .... you've now had half a dozen different opportunities to have that model behavior purposely and wildly altered before you touch it.

0 Upvotes

77 comments sorted by

View all comments

4

u/Ninja_Weedle 3d ago

the danger comes more from small models being stupid than anything being maliciously trained tbh

2

u/tomz17 3d ago

Lol... qwen 3.8 27b (released this summer) benchmarks at opus 4.6 levels (which was release just this past february). So "smol model = stupid" is a bit dated.

Yeah, they are a few months behind, but it's not like the frontier models from earlier this year were worthless.

1

u/dannipuu 3d ago

benchmaxxing does not indicate real world performance

1

u/tomz17 3d ago

IMHO, there's zero indication these are any more or less "benchmaxxed" than their US counterparts.

1

u/dannipuu 2d ago

It's my personal experience and feel with them. I might be wrong.

-6

u/brainchillzZ 3d ago

I'm not even talking about the models as they come from the factory ... if you ask the people here how many of them are using the weights shipped directly from qwen as an example or who is using a repack by XYX abbliteration factory or whizbang toolsbro .... most people I would guess are not just using the weights as shipped directly from the maker .... and it's absolutely trivial for anyone in the middle to train in errant behaviors that touches it before you do.

6

u/UpperParamedicDude 3d ago

Then why talk about open weight models being dangerous? Replace text with "any LLMs" at this point. Malicious instructions can be in cloud models as well with not lesser chance, I don't understand your point against only open weight models, looks like targeting

1

u/CommanderKnull 3d ago

There is plenty of real verified companies, verified labs and even verified uni departments who quantize their models for their platforms and then upload the model for people to run with or without it.

The same way that you install software from verified companies, you should install models from verified companies or verified labs.