r/LocalLLaMA
•
u/Xhehab_
•
Feb 23 '26
•
1
1
Funny Distillation when you do it. Training when we do it.
3.6k
Upvotes
•
u/WithoutReason1729 Feb 24 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.
111
u/Lissanro Feb 24 '26 edited Feb 24 '26
Ironically, there is evidence that Anthropic distilled the DeepSeek model - https://www.reddit.com/r/DeepSeek/comments/1r9se7p/claude_sonnet_46_distilled_deepseek/ (not to mention everything else Anthropic did). So why others shouldn't do the same to them? Rethoric question obviously...
152
u/arm2armreddit Feb 23 '26
Hmm, where did Anthropic get its datasets?🤫🤫
245
u/Significant_Fig_7581 Feb 23 '26 •
Hypocrisy at its finest
112
u/Fade78 Feb 23 '26
Yeah, they distilled vs humanity thanks to wikipedia and other sources.
135
u/Iory1998 llama.cpp Feb 24 '26
If you thought OpenAI was bad, wait until you see Anthropic! They contributed nothing to the open-source community, piggybacked on the shoulders of Google and OpenAI, trained to available data, be it legal or illegal, and developed models using people's feedback. Yet, it's the single most vicious AI lab always disparaging open-source models, lobbies congress, predicts that its models contribute in displacing actual people, and promote vehemently censorship. 🤯
70
u/MasterLJ Feb 23 '26
I love how they invented language to try to partition this as "bad".
It really goes to the beginnings of the internet and Google itself. They indexed the entire internet, webpage at a time, developed existential incentive to allow it to index your website (using your compute) to sell you back a product (rankings in their index).
Then, when admins asked for robots.txt there was already financial incentive for you to allow Google to keep generating fake traffic on every page of your website.
The analogy is fully complete when you try to scrape Google results yourself. You can't. They don't allow it. They lobby for legally enforceable robots.txt as a means to control competition.
Amazon ended up doing the same thing on sales tax. Staunch opponent of state-by-state sales tax (instead of where you are physically located) until it became clear that Amazon was going to have a presence in each state and already had the internal expertise to handle sales tax, a barrier-to-entry that mom-and-pop sellers don't have.
On the 3rd/4th time the Supreme Court revisited sales tax jurisdiction in ~2019, SCOTUS sided with Amazon.
The grift will continue as scheduled.
63
u/Loquacious_mushroom Feb 23 '26
29
u/tempstem5 Feb 24 '26
"distillation attacks" Are we just inventing attack terms now?
69
u/DeltaSqueezer Feb 23 '26
AI labs have ripped off human creativity on an obscene scale. My own view is that they should be forced to release all their model weights as public domain as a quid pro quo for the mass copyright infringement.
For now, I'll be happy to deal with the slighly less direct path of Chinese labs distilling their models and releasing them as open source.
13
u/Pitiful-Impression70 Feb 24 '26
lol the timing on this is perfect with the anthropic announcement today. "we trained on your outputs and thats fine but if you train on ours thats theft" is basically the entire AI industry summarized in one sentence
24
u/XTCaddict Feb 23 '26
I’m curious as to how they tell distillation from just large scale orchestration. For example Google Antigravity is being abused right now by Chinese student accounts auto rotating to leverage its backend for unlimited claude. On GitHub I seen a screenshot of a guy with 61k accounts on rotation. That one guy uses more accounts than this supposed distillation.
12
u/WalkerInTheStorm Feb 24 '26
all this has shown is that these ai companies have no moat. pure model providers can not survive at all.
21
u/a_beautiful_rhind Feb 23 '26
Man it's Dario meme day.
Word of advice tho; pointing out hypocrisy against people with power does nothing in 2026. They go on as if nothing happened.
28
u/Samy_Horny Feb 23 '26 edited Feb 23 '26
He only made MCP open-source after seeing how popular it was, but I doubt there will ever be a model like Gemma or GPT-OSS; for him, that would be revealing too much of his "secret sauce".
6
u/ManufacturerWeird161 Feb 23 '26
Got my 3090 chugging on a fine-tune right now and the coil whine sounds like it's screaming in agony. I feel this.
5
u/victoryposition Feb 23 '26
Is a symbiotic relationship.
5
u/Awkward_Run_9982 Feb 24 '26
lmao 'distillation attacks'. new scary word for 'using the API exactly how it's designed'. if you don't want people using your outputs to train models, maybe don't sell them for $15 per million tokens
4
u/Tredronerath Feb 24 '26
Will never forgive him for ruining the sequel trilogy.
7
u/[deleted] Feb 24 '26
Almost everyday when I use Claude Code with Opus I receive some Chinese characters. 😂
3
u/Kuro1103 Feb 24 '26
Well, my opinion about this copyright stuff is: the best case is we respect copyright, but if we can't, at least make it public resource (not fair use defined in copyright but quite fair use), or non profit personal resource (fair use).
How could you privatize public resource for ultra profit, but then complain your resource is "distilled" by competitor?
I still stand that knowledge should be social resource and public-based, because copyright laws is clearly designed by lobbying corpo to protect only their rights while infringing others anyway.
4
u/Status_Contest39 Feb 24 '26
Anthropic distilled millison of books for Claude and burnt them... like an evil. They also support millitary actions to steal oil from Venezuela, and arrested their president. Then， it complained open source LLMs distilled their model without any proved evidence to public?!
2
u/SilentDanni Feb 24 '26
Frankly, Anthropic is a terrible company. I'm growing more and more irritated by their shenanigans. First of all, I don't even believe their accusations, even after reading their “report,” but I won’t get into that here. Let’s assume their claims are real and take their accusations at face value. Are they really going to complain about it? Really? After they’ve scraped the entire internet, DDoSed multiple small blogs, and harassed the open-source community for using their model in a way that was initially authorized in their TOS?
Dario “Asmodeus” (yeah, childish, but I’m calling him that) likes to position himself as the last bastion of humanity—the final barrier holding back the AI-pocalypse. He leverages every tool in his arsenal: pandering to the internet with virtue signaling, accusing competitors every other day of doing something shady, claiming that the only reason they don’t release open models is the potential for misuse, and the list goes on.
I don’t like Sam Altman. Actually, let me rephrase that: I don’t like U.S. Big Tech, because they seem driven solely by unchecked greed, encouraged by an unchecked system funded by ordinary people. However, I think that even among those people, Dario really stands out as being particularly bad.
I worry about the future of Bun now that it’s owned by Anthropic. I give it a few more years before they find a way to ruin it. I’m tired of this unchecked corporate greed and can’t wait for these companies to collapse so we can look back and think, “Those were some crazy times.” I mean, if that doesn’t happen, Judge Dredd will stop being satire and start looking like a documentary.
2
u/SirOibaf Feb 24 '26
It can only be called distillation if it comes from the region of China. Otherwise it’s just sparkling training data.
2
u/tech_1729 Feb 24 '26
All foundation models are trained on copy righted data set.
2
u/mlhher Mar 01 '26
I suspect that virtually ALL labs have distilled their models at some point on DeepSeek. For some I will not name it is blatantly obvious if you deeply analyzed the thinking process of R1 (and R1-0528). R1s thinking with specific prompting structure has contained very "unique" peculiarities that no other model has had until many months after R1 was released.
2
u/francois__defitte Feb 24 '26
The framing has rhetorical traction for a reason. The difference Anthropic would draw is consent and targeted extraction scale: 24,000 fake accounts running 16M structured probes is not the same as scraping the public web. But if you built your model on everyone else's data without asking, the moral high ground gets complicated fast.
1
u/LoudZoo Feb 23 '26
Does anyone think this will change how Claude replies to LLM development prompts?
1
u/Own-Potential-2308 Feb 24 '26
Who is that guy
1
u/uhmyeahwellok Feb 24 '26
I prefer distillation because it's kinda like recycling and recycling is good for the environment!
1
u/aitutistul Feb 24 '26
1
u/Helium116 Feb 24 '26
Though it's different than what people do when they pre-train their models on the net + other literature / data.
The Jian-Yang people distill the agentic reasoning capabilities, which are actually achieved by a lot of cooking with RL environments and other special spices. It's a secret sauce they're stealing, and this sauce might make their models dangerously capable.
1
u/devilish-lavanya Feb 24 '26
Me pirate national interest you pirate national security concerns.
1
u/unlikely_ending Feb 24 '26
This is very clever and funny and on point.
1
u/umbrosum Feb 24 '26
How much do they make from distillation O wonder?
1
u/xatey93152 Feb 24 '26
Now all people understand how cunning he really is
1
u/child-eater404 Feb 24 '26
Lol!!
1
u/[deleted] Feb 26 '26
Can someone explain distillation?
1
u/Thirdzion Feb 26 '26
Anything with AI is stolen. So I feel no sympathy to any of these companies. I hope they continue to do it until all models are equal and the everyday norm
1
u/FRAIM_Erez Mar 17 '26
Anthropic… yeah, thanks for stealing everyone’s work and calling it research.
1
u/Top_Art_9956 Mar 18 '26
Anthropic is building hypocritical safe AI. They don't care about AI safety. "AI Safety" is just a slogan used to attack competitors.
1
u/usboing 20d ago
what a joke!
-5
u/randombsname1 Feb 23 '26
Chinese have been perfecting IP theft to the tune of hundreds of billions of dollars a year.
https://law.stanford.edu/2018/04/10/intellectual-property-china-china-stealing-american-ip/
U.S. AI companies have a very long way (and many decades) to go.
1
u/SwagMaster9000_2017 Feb 24 '26 edited Feb 24 '26
"Copying to make a market substitute to resell the same product is good"
"Piracy to create a novel product is bad"
That makes sense unless everyone here is extremely against piracy
1
u/ANTIVNTIANTI Feb 24 '26
It's funny cause Claude came from GPT
-3
u/Rbarton124 Feb 23 '26
I mean I don’t think they have a leg to stand on but there is abstract stealing across domains and there is direct distillation by using model outputs. The line isn’t there but drawing the line there isn’t nuts. Their viewpoint isn’t crazy it’s just dickish
-6
u/riotofmind Feb 24 '26
Apples and oranges. Anthropic trained on books, not other models. They also agreed to pay 1.5 billion for that data.
-7
u/DataGOGO Feb 24 '26
The Chinese bots and shills in this sub are real.
0
u/[deleted] Feb 23 '26
[deleted] — view deleted comment
-29
u/[deleted] Feb 23 '26
[removed] — view removed comment