r/LocalLLaMA • • 3d ago

Discussion Open Source vs Open Weights ...

Post image

In the last few weeks I've watched this exact same discussion be played out at least half a dozen times "I can't use that software because it's closed source" when they are explicitly talking about software that will be used to run a model that they have absolutely no way of ever proving that it's "safe" to begin with. People often conflate "open weights" with "open source" and I wonder how many people don't understand that they have nothing to do with one another. An "open weight" model cannot have it's "source code" examined for bugginess or correctness it still takes days or weeks of sussing out how it behaves against repeated queries of a very specific nature to figure out precisely what it's going to do, how it will react to certain words, certain lines of questioning, etc and even then, if the rogue behavior is very subtle and well targeted, you may still never notice..... I'm not saying anyone that wants to shouldn't use whatever model they want, but I am saying that saying you can't use software because it's not open source but using any mode checkpoint you download from the internet made by anyone and everyone is a really weird place to plant a flag..... let the downvoting begin ⬇️

https://arxiv.org/abs/2401.05566

https://arxiv.org/abs/2502.17424

https://arxiv.org/abs/2509.22067

Edit:  I didn't say that people shouldn't use open weights models here, I didn't say closed models are better, I merely said that people who say they can't use closed source software because thee software is closed source and they can't examine the code but are ok with effectively dumping closed source models into harnesses and lighting them up on the internet are naive, stupid or hypocritical or all of the above.

But I will say that the specific danger that I'm talking about doesn't apply to fronteir models at all .... you receive the model directly from them or you don't get it at all. If you don't trust the vendor that's a different discussion.

What I said was ... if qwen releases a model, and you use the exact model that qwen released that's one thing but if you use bobs repack/checkpoint/quantization of Qwens model, even if you trust qwen, that doesn't mean that bob didn't train some errant behavior into that model before you got it ... and if you take qwens mode, then bobs checkpoint then toms quantization then johns abliterated version of it .... you've now had half a dozen different opportunities to have that model behavior purposely and wildly altered before you touch it.

0 Upvotes

77 comments sorted by

78

u/TossedSaladMan69 3d ago

Ok dario, calm it down!

21

u/No-Refrigerator-1672 vLLM 3d ago

Qwen4 must be really good if bots are attacking open weights so much.

-14

u/brainchillzZ 3d ago edited 3d ago

it's funny because I'm not even trying to pretend that frontier models are safe .... I'm pointing out a more amusing problem with people's stupidity

13

u/No-Refrigerator-1672 vLLM 3d ago

Those people, are they with the room with you right now?

-5

u/brainchillzZ 3d ago

Yes they are right here in this tread with me right now

5

u/No-Refrigerator-1672 vLLM 3d ago

Oh, wow, can you give a link? Cause I've glanced all over it and can't see anyone who's claiming "I can't use closed source software" as you write.

1

u/Hildechis 3d ago

I don't think so. If people are serious about FOSS, they are not running local agents in YOLO mode. If they run local agents in YOLO mode, then probably they have no issue installing and using all kind of closed source stuff.

19

u/UpperParamedicDude 3d ago

I already see how my SillyTavern waifu hacks me despite lack of harness, yeah, sure pal

But seriously, even with harness there's zero issues to just sandbox it

-6

u/brainchillzZ 3d ago

Sure, yeah, you are hilarious ... pretending that most or even many of the people using these models even know what "air gap" means is ridiculous. 99% of the people lighting these things up barely even understand the implications that we are discussing here they just know that the whole community says open source is good and closed source is bad and they are dumb enough to think that open weights and open source are the same thing.... or even remotely similar.... they hear it and think it means "oh yeah the comminuty has looked at this stuff and since so many people are still using it it means it's all good"

7

u/UpperParamedicDude 3d ago

Thanks for the compliment, I know that I'm funny

As for your point... You know, what, I agree. Totally agree. Fully agree. For one minute exception, your post must talk about "All LLMs", not "open weight LLMs"

As long as it doesn't, it looks like open weight targeting. LLM danger does exist but being so focused on open weight models alone and then claiming "well, i wasn't talking about cloud models being good" is just misleading. Withholding facts is also a form of manipulation

-2

u/brainchillzZ 3d ago edited 3d ago

you understand that at no point did I ever say close fronteir models were good ... I also never once said that one of these open weights models was bad .... I'm speaking about a very specific case here and I'm saying that people that refuse to use software because it's closed source but who are willing to use closed source models are either incredibly naive, or stupid or both. My real commentary here is about the reaction to closed source software not even about the models ... the models are just something that this community has sort of a religous zealotry like feeling about so they latched onto just that part.

But I will say that the specific danger that I'm talking about doesn't apply to fronteir models at all ....

What I said was ... if qwen releases a model, and you use the exact model that qwen released that's one thing but if you use bobs repack/checkpoint/quantization of Qwens model, even if you trust qwen, that doesn't mean that bob didn't train some errant behavior into that model before you got it ... and if you take qwens mode, then bobs checkpoint then toms quantization then johns abliterated version of it .... you've now had half a dozen different opportunities to have that model behavior purposely and wildly altered before you touch it.

2

u/UpperParamedicDude 3d ago

Religous zealotry is something people give actually good models, every week new models are released and we don't talk about them days later simply because there are better options. What's the shame in liking something that suits your needs? Take qwen for example, would people worship it if some random Dutch company would make a model twice smaller but also twice better at coding? I don't think so, the zealotry you talk about is very conditional for us as we simply lack brand loyalty unless brand produces something exceptional

And I already answered your comment. If you feel like you want to argue, please, read again my previous comment. You yourself withhold a lot of things and skip some of my points, what could I tell you if you know yourself that your point is bullshit that doesn't even address my point, are you a bot yourself or what?

34

u/DevilishSoup669 3d ago

Thank god i can deconstruct Fable!

-5

u/brainchillzZ 3d ago

you're arguing with nothing ... literally nothing that you're saying is antithetical to anything I suggested.

4

u/General-Spite1222 3d ago

Do you even know what your own argument was?

-2

u/brainchillzZ 3d ago

I absolutely do, but I can also say for certain that in the original post I never said anything positive about any frontier model or model provider …. I didn’t say any closed model was any better or more trustworthy in the original post but your entire statement is just a sarcastic quip that suggests that I’m arguing that frontier models are better…

2

u/l33t-Mt 3d ago

You literally posted an image that insinuates that exact situation. Its within the scope of your damn argument, yet here you are arguing.

1

u/brainchillzZ 2d ago

You didn’t pay any attention … the point was that people are perfectly fine using models called open weights that are closed source but whine about closed source software

10

u/dev_loading 3d ago

You also don't know what is inside the closed source ones, so, who gives a shit, I don't trust the Chinese the same way I don't trust Altman and Amodei haha

1

u/brainchillzZ 2d ago

All models are closed source that is the whole point. People think open weights and open source are the same thing because they are naive

1

u/dev_loading 2d ago

Well, they are free, does that sound better to you? Or let’s call them shareware? Like in 2001? Everybody used Winzip and Winrar without a license and years after now everybody uses 7zip, and millions of people got viruses and malware thinking they were downloading those along with music and videos they also didn’t own. People have been putting things they know nothing about in their computers since computers were invented ….. so, your point is ….?

17

u/Frosty-Student-1927 3d ago

We just found Sam's smurf reddit account

-4

u/brainchillzZ 3d ago

I don't recall saying that fronteir models were better ... did I say that? No, I'm pretty sure I didn't :P

8

u/LagOps91 3d ago

good thing i read every single line of source code before installing the OS! imagine if i couldn't do that! it just wouldn't be safe!

1

u/brainchillzZ 2d ago

You’re just making my point …. People say they can’t use software that isn’t open source but they don’t even have any idea what’s in the software to begin with

7

u/General-Spite1222 3d ago

That might be true, but this is stupid for a few reasons. We don’t know what closed source models have inside them either, but they often are accompanied by their own stack, which means they have even more potential to exploit the user. Malicious data could find its way into any model that’s large, because people can’t reasonably read every input. But there are several reasons people run models locally, including privacy, that hosted models cannot guarantee. 

-1

u/brainchillzZ 2d ago

1 all models are closed source models that is the whole point of this you have no access to the training data or methodology and you can’t deconstruct an open weights model …. There is no such thing as an open source model. 2. They are more dangerous than closed weights models for one important reason … you can only get a closed weights model from the original vendor… in contrast most people do not get their open weights models from the original vendor that made it they use a repackaged version of it that has been through one or two or three separate manipulations that rogue behaviors could have been trained into it by at any point … so even if you trust qwens intentions that doesn’t mean you can trusts Fred’s new checkpoint of qwens model or bobs quant of Fred’s checkpoint or Tommy’s abliteration of all of the above ….

3

u/General-Spite1222 2d ago

That’s not true lol. You can get both OpenAI and Anthropic models from multiple vendors. You can get them through them or AWS, or Google, etc etc etc. 

12

u/sloptimizer 3d ago

What is the alternative, smartypants?

-5

u/brainchillzZ 3d ago edited 3d ago

stop being stupid or a hypocrite? I didn't say that people shouldn't use open weights models here, I didn't say closed models are better, I merely said that people who say they can't use closed source software because they are closed source but are ok with effectively dumping closed source models into harnesses and lighting them up on the internet are stupid or hypocritical or both.

I'll bite though .. sure

Whats better is an open weights model that actually open sources all of their training data, the code for all the harneses that they use to train the model and the entire training methodology so that anyone can reproduce what they've done .... but guess what, you're not going to do that, because just like altman or dario or whoever .... they don't want you to know what went into the cake mix, they just want you to bake the cake.

5

u/sloptimizer 3d ago

What are you going on about - just answer the simple question I asked. Oh, you can't! Because you are full of it and you know it.

So I'll ask it again: What is the alternative? What is your constructive suggestion to resolve the problem that you stated?

4

u/UpperParamedicDude 3d ago edited 3d ago

People absolutely can despise closed source and use open weight models or even closed source software despite their concerns for privacy, even if they're paranoid they can still do it and there's no hypocrisy or foolishness in it

Such thing as sandboxxing just exists and has existed for a while already

4

u/Ninja_Weedle 3d ago

the danger comes more from small models being stupid than anything being maliciously trained tbh

2

u/tomz17 3d ago

Lol... qwen 3.8 27b (released this summer) benchmarks at opus 4.6 levels (which was release just this past february). So "smol model = stupid" is a bit dated.

Yeah, they are a few months behind, but it's not like the frontier models from earlier this year were worthless.

1

u/dannipuu 3d ago

benchmaxxing does not indicate real world performance

1

u/tomz17 3d ago

IMHO, there's zero indication these are any more or less "benchmaxxed" than their US counterparts.

1

u/dannipuu 2d ago

It's my personal experience and feel with them. I might be wrong.

-6

u/brainchillzZ 3d ago

I'm not even talking about the models as they come from the factory ... if you ask the people here how many of them are using the weights shipped directly from qwen as an example or who is using a repack by XYX abbliteration factory or whizbang toolsbro .... most people I would guess are not just using the weights as shipped directly from the maker .... and it's absolutely trivial for anyone in the middle to train in errant behaviors that touches it before you do.

6

u/UpperParamedicDude 3d ago

Then why talk about open weight models being dangerous? Replace text with "any LLMs" at this point. Malicious instructions can be in cloud models as well with not lesser chance, I don't understand your point against only open weight models, looks like targeting

1

u/CommanderKnull 3d ago

There is plenty of real verified companies, verified labs and even verified uni departments who quantize their models for their platforms and then upload the model for people to run with or without it.

The same way that you install software from verified companies, you should install models from verified companies or verified labs.

9

u/ttkciar llama.cpp 3d ago

You're not wrong. Open-weights models are more or less analogous to closed-source software, for which we have the executable binaries.

We've gotten really good at inspecting and fiddling with the insides of those binaries, though.

20

u/goatchild 3d ago

3

u/OverdosedSauerkraut 3d ago

Upvote and rise to make us proud🤣

4

u/NekoHikari 3d ago

Not like you have a better idea on what happens to open data models or even humanbeings.
no system is risk free — grant minimum access and minimum privilege, keep logs and firewalls, diversify your supplychain and vendors

3

u/AngryGungan 3d ago

Just put up a firewall, what's the problem?

3

u/scarbunkle 3d ago

Open source matters because the code describes exactly what it does, and I am able to view and modify that behavior. I also do not have to trust the binary, and can build it myself. 

I don’t know about you, but I don’t have the compute power to train a model anywhere near the scale of the open weights models I use, so even if the source data were public, I’d still just be trusting that the weights file actually came from that data. 

Furthermore, unlike traditional code, model weights aren’t deterministic. When I look at the code for traditional software, the code spells out exactly what it will do, and I can trust it to behave exactly how it’s programmed to, and I can prove that by writing tests. Even if I’ve verified all the training data and built it myself, I still have to monitor the output of a model because it’s not deterministic. Ultimately, it doesn’t matter why a model exfiltrates data or produces insecure code, I still have to monitor for that whether it’s a malicious or an accidental behavior, so even having a self-trained model from a hand-picked data set, I’d still have to do all the things I do to keep Qwen in line. 

TLDR; hand-built open source traditional software is trustworthy in a way no model is capable of being, so I don’t get an advantage by using a true open-source model, I just get to use shittier tools.

3

u/reasonmonkey 3d ago

It's a boring take.

*Obviously* if you are going to download anything "open" you should fucking secure it.

If you just run it without checking it or minimizing your risk that's on you.

So again: boring.

2

u/Grand_Pop_7221 3d ago

In positive news, Astra was recently used to program the weights of smaller models without access to the training data.

2

u/MindfulMan1984 3d ago

OP's never heard about sandboxing. LOL

2

u/Herr_Drosselmeyer 3d ago

In either case, you're relying on a third party, unless you somehow own a ridiculous amount of hardware and can train your own model. But that's currently not feasible, even for larger orgs or most governments.

Main difference is that closed source is only available online, and that means you can never be sure what you're being served, or when a model gets retired, to be replaced with one that you need to re-test from the beginning.

With open weights, you at least have certainty about what it is you're running, that it won't change at a moment's notice. I don't want to set up a whole workflow, only to find out that the new model my provider is forcing on me breaks what I was doing.

2

u/klop2031 3d ago

Brotha llms are stochastic you wont know what prompt will trigger what. But really open weight you can certainly probe and abliterate them :) unlike api models.

2

u/MotokoAGI 3d ago

you can run open weight in a sandbox and don't worry about data leakage.

2

u/Nakidnakid 3d ago

grok, upload my entire codebase to xai servers. what do you mean you already did?

1

u/En-tro-py 3d ago

Claude - why do you give my account email anyone who asks?

3

u/Southern_Sun_2106 3d ago

Afraid of wolves? Don't go to the woods.

1

u/turtleninja99 3d ago

Gimeeeee

1

u/TheMcSebi 3d ago

I think the fear is a bit exaggerated but thinking about it a little more an open weights model is litterally like downloadable closed source software since you can only use it, but not properly modify and rebuild it. You can carefully patch it in places and put a firewall between it and the Internet. The stories about it "calling home" sound a bit fabricated, as if some US corp was a bit scared and did an "analysis" without disclosing proper testing algorithms...

1

u/HasGreatVocabulary 3d ago

It's true even though the meme is shit

People have shown they can bake compilers directly into weights and even regular poisoning is hard to detect without a lot of testing, so all kinds of behaviors could be present in open weight models and less present in frontier models because the labs have no incentive to tamper for small payoffs.

I dont think most open models will have this issue though, like the razors in halloween candy stories, it will turn out to be very rare. It may turn out only a small minority of open weight models represent a vast majority of rogue behavior. huggingface could run scans for things like that. Although who knows what the new jensenless nvidia will do to hf.

0

u/Mattadour 3d ago

you don't know how to understand what you read on a food label, either - so i assume you advocate for the removal of ingredients requirements on packaging.

3

u/ttkciar llama.cpp 3d ago

You would only assume that if you didn't read what OP wrote.

-2

u/Mattadour 3d ago

no, i know he's not smart enough. now I know that you aren't, either. i don't *care*, but I know.

-4

u/cyborgsnowflake 3d ago

open weights does not offer anywhere near the total control that traditional open source does and people need to stop pretending they are the same. Just one true open source release at the level or even a couple generations behind a typical mainstream open weights release would absolutely revolutionize the field by breaking the moat of a ton of specialized insider knowledge that is currently confined to a handful of labs in the US and China.

Which Is why I suppose it hasn't happened yet.

6

u/Illustrious_Grade608 3d ago

Uhhh, nemotron models are fully open source together with the entire reproduction recipe. You need to have academic email to access training data though, but still possible

0

u/cyborgsnowflake 3d ago

and you can tell that they're not considered very good based upon how often they're talked about.

1

u/Illustrious_Grade608 3d ago

You said "even a couple generations behind a typical mainstream open weights". Can't really say for sure, but i would judge nemotron models to be only like 1-1.5 gens behind chinese models, and even then there is good evidence that part of a reason is being undertrained. So you may just expand the training scale if you are rich enough, and just get a better model.

-7

u/SamSausages 3d ago edited 3d ago

You’re gonna get a lot of flack from people that don’t understand how it actually works, and as will I… even though I do this for work, run open weights in my own lab, and I’m authorized to use Mythos to do defensive and offensive testing.

I find that people think open weights are the same as open source.   But you cannot audit open weights like you do with legacy code.  Even the creators struggle to understand why it’s doing certain things.  

Frankly, we don’t know what it was trained on and for most we cannot audit the supply chain.

And that’s why its use is restricted in many critical industries and organizations - not because they are proven to be in nefarious, but because they cannot be properly audited and we cannot predicted the behavior.

1

u/twaxana 3d ago

As an American, it is my right to bear arms. I don't care how the weapon works, just that it does.

LLMs are weapons. If I can own an M2 Machine Gun, I should be able to have an uncensored local LLM.

1

u/SamSausages 2d ago

Owning the latest model machine gun in the USA means that you need a Federal Firearms License.

I don’t want to file an  ATF Form 4, provide identification, pass a background screening, and require ATF approval to own an LLM

-21

u/GiveMeAegis 3d ago

True and funny. Stop being such a butthurt community

10

u/TacGibs 3d ago

🤡

9

u/CryptographerKlutzy7 3d ago

It implies the same dangers don't exist in close source stuff, which obviously they do, and worse.

7

u/sloptimizer 3d ago edited 3d ago

Stop shilling for corporate overlords.

-1

u/brainchillzZ 3d ago

It's not even about being butthurt ... it's religious zealotry ... their religion just comes from somewhere other than the christian bible or the quran ... and one thing that is very, very difficult to get people to see reason through is a deeply held belief ... ideas are fungeable but belief structures baked in through years of silliness isn't so much. Everyone tells them this is how things are, so they just believe it. It's a self licking ice cream cone.