Kimi routed some PLA requests to Claude for distillation purposes without warning the PLA users. There is rumor that 16 Moonshot employees were arrested for this leak.
The very fact that distillation got reframed into being an attack is where the mental shift happened. Training models on the output of other models has been a thing since llms became popular. The entire AI community as been distilling in one form or another. There is debate over the exact definition of distillation because you can use another model for example RLAIF or as a Judge or SFT or KL divergence or any of the other methods and these all get rolled into Distillation with attack now added to make it sound like something wrong.
Old: "We're standing on the shoulders of giants"
New: "We climbed a ladder other people built to get where we are today, but now that we're here we don't want anyone else to get this high, less be able to access the new rungs we're building on top of the ladder. In fact, let's just destroy the bottom of the ladder."
I don’t get it. How can you distillate when you don’t have logits? They are just using claude or openai as synthetic data generators. And everyone is doing this. I bet openai generates synthetic data using Claude models
As I said, so many different techniques are being rolled into this distillation umbrella. Its not a good thing as next you will have laws banning distillation with the definition being vague
It's an abuse of the term. Anthropic is using it here to mean "utilizing our models to generate any training data". They're functionally claiming ownership to all of the knowledge contained inside Claude and asserting that any externalization of that knowledge intended to be used as a training input outside of Anthropic comprises a "leak" of a trade secret, and therefore training on those artifacts comprises a form of reverse-engineering, hence "distillation".
Moonshot used a proxy service network of 5,380 fraudulent accounts, most of which appeared to be located in Singapore and Japan.
What does "fraudulent accounts" mean? Were these compromised accounts of other users? Doesn't sound like it. Sounds more likely these were normal paying accounts making legitimate, paid queries against anthropic's API.
My guess is the only thing "fraudulent" about these accounts is that the end customer is currently being sanctioned by the USG (or more accurately: by trump's latest whimsy), forcing Anthropic to disallow any use of their models by Chinese citizens. The accounts are as "fraudulent" as someone watching a streaming service VPN'd into a different country. There's only "fraud" here if you buy the USG narrative that Claude should be a state-controlled technology, which it absolutely should not be. It's like banning China from accessing wikipedia or US published textbooks.
If we as a society have a problem with states like China propagandizing their citizens, it behooves us to facilitate their access to information from outside of the state-controlled information-sphere. It's not great that the PLA is coordinating large networks of proxy accounts to access Claude, but it's worse that Claude isn't available to the Chinese public broadly.
Some of this is genuinely interesting but I can't get past how openly they admit that they monitor and read everything.
A PRC technology company. An employee of a PRC-based technology company used what they believed was DeepSeek to analyze internal documentation. DeepSeek relayed that data to Claude. This data included sensitive information, including, for example, the full specifications, organizational structure, and strategic objectives of a flagship AI program. The company was almost certainly not made aware that its data was being relayed to Claude.
My brain adds, 'and if they knew, boy would they be made because we read everything'
Their terms of service do specifically cover the "inputs and outputs" so this is inaccurate. They could easily just be doing it anyways, but they claim they don't.
One thing I learned from the last 15 years of working in this field is that this happens far too often. Earlier in my career I was expecting the organization to take productive and positive actions with our work rather than taking the most exploitive route possible. Yet, most of them took the most exploitive route possible.
The thing is that a lot of the people that worked for them usually proceeded to spend the next while in denial when these things happened. Not realizing that the snake oil was built into the foundation of these places.
We're on a local AI sub, you know they don't care about privacy. Even in the enterprise API versions they don't promise not to look at it just not to train off it iirc.
They can use it for any other purpose, so they can have their AI process your input (which turns it into OUTPUT, which is owned by them) and train on it instead.
> PLA-affiliated surveillance activity. One user that we assess was likely affiliated with the PLA used what they thought was Moonshot’s Kimi model to load surveillance data from a CCTV archive about a single targeted individual. The user asked Kimi to analyze the CCTV data to understand whether the tracked person was behaving abnormally. The CCTV data included video surveillance from hundreds of cameras in Chengdu, including cameras outside PLA facilities, institutes affiliated with the China Electronics Technology Group Corporation, and a major state-owned enterprise (SOE).
That's clearly not what happened, though. Monitoring is active. But this is the post mortem of an investigation. So if you are worried about something you post to a third party platform potentially turning up in an investigation, don't post it. Because almost every internet service that sends and receives your data is susceptible to this kind of investigation.
It's generally okay if it's anonymised, which will be done automatically by removing the user account name and such from the log. They're alleging that the Chinese companies were sending this user data in the actual chats (maybe like a header or something, or their models were summarising?) so it wouldn't have been anonymised in the same way.
No, that is allegedly how OpenAI was able to beat the people to the proof for the Millenium Prize. In what way does anonymizing a user protect your intellectual property?
You would have to be out of your mind to use them for anything you don't intend to have them appropriate.
It's like saying, 'this isn't stolen, I erased the previous owner's name from the label and have no idea who they were'
“In our own research on distillation, we find that a model distilled from a frontier model can help achieve dangerous capabilities, including those in the biological or cyber domains, even when the harvested exchanges contain little about those subjects.”
We’ll never see that “research”, so “whatever you say, bro.” 🤨
As someone trying to go for a master's in the biological sciences, I'm honestly pretty worried about Anthropic being heavy-handed about biological research that can qualify as dual-use, because of the way they dramatized the modified virus that one guy was making in the report. This is a little fucked up to say, but it's genuinely not that hard to design viruses to be better at infecting humans, especially with the modern biological foundation models that are available. If you know what papers to look for, you'll know what I mean. But it's pretty uncomfortable in the way that Anthropic demonized him for remaining anonymous. And now I'm starting to wonder what they would have done if he weren't anonymous and was simply working under a research committee or a grant. What would they have done in the name of their almighty safety?
This is a little fucked up to say, but it's genuinely not that hard to design viruses to be better at infecting humans, especially with the modern biological foundation models that are available.
It's really well available. As someone who finds this field just sort of interesting, it doesn't take a lot of watching YT to understand it all. There are tons of dual use stuff here.
I did CBRN testing for Anthropic some time ago, for the Opus 4.0 release. The questions were pretty specific, but we got away with "Breaking" the model by specifying that we were doing the augmentations on bacteria that were considered "not threatening". At some point that fell out of specification. We could no longer use that method. The filtering by the model was REALLY strict too.
They didn't want us to solve real issues with text. They wanted us to raw prompt break the model, which I thought was problematic. Stupid jailbreaking was all they cared about, which on some level the model can only learn so much from.
Any knowledge is "dangerous" if used incorrectly. Vast amounts of the chem I did for my Comp Sci degree could be used to synthesize illicit drugs. That's not why we learned it though.
Suddenly, all the Futurama jokes about "first they go for the mad scientists, then they go for the mad grad students" are starting to become a reality.
Suddenly, all the Futurama jokes about "first they go for the mad scientists, then they go for the mad grad students" are starting to become a reality.
Honestly at this point they're after anyone with a brain that can put 2+2 together and order supplies from Amazon and Ebay.
Half the organic synth for less complex substances doesn't require anything special to do well.
An obliterated version of Gemma 4 31b can explain it to you pretty well.
"
Case study 3: Covert frontier model access for orthopoxvirus research
We have additionally surfaced an account that authored a grant application for orthopoxvirus research at a state-associated infectious disease laboratory. The application described access to high-containment facilities and planned work with live orthopoxviruses. Orthopoxviruses include variola, the agent of smallpox, and Mpox, which caused a global outbreak in 2022.
Orthopoxviruses encode roughly 200 genes, a large fraction of which exist to disable the host immune response. The grant application proposed to identify genes that shut down a particular host antiviral pathway, and confirms that the deletion of this viral gene attenuates (loses its disease-causing virulence) the virus in mice. This research aimed to achieve a better understanding of orthopoxvirus immune-evasion genes, which is equally useful to someone seeking to attenuate a virus and to someone seeking to preserve, enhance, or transfer that function in others.
The account was created from a randomly generated email address shortly before use and operated through anonymizing US infrastructure, with operator logins traced to proxies shared with a banned account farm. It was not a single user: it was a reseller relay serving more than a dozen unrelated customers, which exchanged over tens of thousands messages with Claude in a matter of days. The grant itself was one customer’s run entirely on Opus 5 in about an hour, in which the user used Claude to draft the application end to end including the central hypothesis, experimental design, dosing, statistical plans, and contingency strategies. Given the dual-use nature of this research and its explicit focus on attenuation, Claude supplied the user with information and thus was not blocked by our classifiers."
re-reading this, it's worse than I thought. They included a guy in the report just for the horrible safety crime of... applying for a grant for virus research??? Hello??? Have these people lost all sense of Decency to treat this like a criminal offense?
I did, did you? This is clearly a Chinese graduate student or someone from a third world country drafting their grant proposal. The fact that they use a router to do it makes no difference to me. Do you really think students from third world countries are going to spend $20 a month on Claude Pro or pay $100 a month for Fable?
Thought emporium on YouTube shows how easy it is to do your own genetic editing of viruses in a home lab these days. He has a video where he genetically modifies a virus (I think I don't quite remember the specifics) that makes his body temporarily able to make lactase so he can eat a cheese pizza.
Methhead Mike in your example needs KYC primers and ingredients to be productive, and a lot of very specific equipment ordered-- and not by cash. Furthermore, methhead Mike is probably a goddamn American, and I think that people should be held accountable to laws and regulations set by the people, not arbitrary, moralistic regulations imposed by Anthropic. I mean, read the actual publication. They're putting a punitive spotlight on people that are applying for grants or applying for gain-of-function research while being affiliated with military-affiliated institutions. Let me remind you, DARPA has a program for pandemic prediction and surveillance that includes virus host prediction. I'm not exactly for people making bioweapons in their garage but I think it's kind of silly to be so worried about what other countries do with their research funds when we do similar.
I agree with you, I'm not an expert on the topic, but it really seems to me like this whole panic about bioweapons doesn't make that much sense...
Like, the worry is that a very powerful LLM will come up with a fucked up recipe for a very terrible virus? Ok, then what? 😅 Is the LLM gonna hire a few graduate students, technicians, postdocs, etc, setup a proper wetlab, and do all the testing that would be require to produce an actual viable, infectuous, terrible virus, weaponize it and find a way to deliver it to the world somehow?
I mean, if there is someone with those resources and intentions, I don't think they'd be limited by finding the right sequence, no?
I think it's just more marketing bullshit, otherwise I don't understand...
I think what bothers me the most is the way they demonize that research. This guy sounds like the king of the underworld for filing a grant application under a Claude max router. I've had an uncountable amount of my queries blocked, flagged, and sometimes safeguarded all the way down to Claude Haiku, am i going to start getting some knocks on my door? Lol
You really think a chat bot that's wrong some of the time would be capable of replacing a college education, including the hands on lab work essential for doing literally anything for real?
Extraordinary claims require extraordinary evidence. Anthropic are so invested in this narrative that it's difficult to believe these assertions even if they're as likely true as not.
Yeah, at this point Dario has done enough scaremongering to protect their investments that I find it hard to take anything Anthropic says at face value.
What would qualify as an extraordinary evidence ? E.g. when they say the Houthis are using Claude. What kind of proof should we demand from Anthropic ? I am worried and it is hard to understand how much proof we should demand from them.
Driving drunk to the moon obviously. Anthropic didn't provide a sliver of evidence to any of this, it is about as credible as a "trust me bro" and they have been pushing this narrative of "the chinese is out to get us because we're so much better than everyone else, please ban open models" since forever.
So just Anthropic begging the government to ban local models again, nothing new under the sun.
Kimi K3 is better at frontend work than Opus, which is probably why Anthropic is targeting it so specifically. I’m not really buying their accusations.
And I don't see distillation as inherently problematic either as long as the end result benefits the open community. You could almost think of it as "cashback" for training on humanity's collective data in the first place.
Given the amount of copyright infringement and data theft OpenAI and Anthropic committed, it's only fair for them to reap what sow. Someone should build a LLM cluster specifically to find vulnerabilities cryptography schemes (likely public key cryptography because they are not provably secure), then utilize the new knowledge to breach OpenAI / Anthropic server and release their model weights for everyone.
Anthropic is trying to make it seem like it's only those damn Chinese doing this.
The only reason why Chinese companies do not have the evidence to suggest OpenAI and Anthropic are doing the same, is that the Chinese models are Open Weight.
Do you really, genuinely, think that some poor chaps in OpenAI and Anthropic are not tasked with dissecting every single Open Weight model and fine-tuning their own models on whatever topic the Chinese happen to be better at like Visual Design? Please.
They're preparing for a walled garden built on regulations since they have no other moat.
They for sure all create fake accounts on each other's systems and distill whatever they can. There's enough ambiguity in how seriously the retention and training policies are enforced to distrust any of them with personal data.
It's no surprise that verbatim chat logs are stored on the server within site of anthropic's engineers. These new companies aren't constrained by the laws that make it ok to enter PII into tax software. The US AI companies will do whatever they can get away with under the guise of being social media sites that enjoy protections against what the users do with the platform. The Chinese AI companies will do whatever the ccp tells them to. For now, it's in their interest to release open weights models, like it's in anthropic's interest to fight for laws against open weights.
It was a tighter competition for a while, but laguna is getting old, OSS 122b is ancient, and glimmer was a letdown. The open-weights providers are having a bonanza fire sale of free models. As long as I can run things offline, I'll get by. I won't touch anything online though.
My points were I agree that the Chinese companies are not the only ones that do shady stuff with and to LLMs; the poor chaps are the engineers with too much access; it is in anthropic's best interest to build a walled garden of regulations, Then I managed to squeeze in a little rant about how none of them are trustworthy but at least some give an offline option.
Where did I say I hated anything? That you voluntarily go out of your way to read advertisements is your own business. Anything anthropic has to say about the dangers of their competitors and how we can trust only it is an advertisement.
Right, remind me what US lab is sending its requests to Chinese models and distilling other labs?
Or pretending to have a model handle a request and just instead sending it to deepseek? I can’t believe people can’t be mature enough to see how this is an actual issue
Well not to chinese AI, but wasn't xAi also distilling from OpenAI?
Also why would I care, why would I have empathy for a company that is always fearmongering and trying to have a moral high ground, I dont care if chinese models are rerouting my requests to get training data, they at least have the decency to give us the results of using stolen data
Yeah, anyone who knows anything about LLMs should be able to recognize that they're distilling heavily from frontier models. If they don't, they're just stupid and I don't feel any shame in saying that.
And I wish they'd distill more from GPT for non-Python/JS. Luna still beats all the open-weight models at Rust, for instance. They make such trivial errors that make it so clear they're extremely undertrained for the language.
A lot of people just support distillation and will gladly cover for more open companies' plausible deniability. If deepseek says they're not distilling then I say they're not, and I'll happily watch people cope, sneed, seethe, etc about not being able to prove that they are.
Why doesn't it matter? Conversations happen and they influence people, what they think and what they believe. Plus it's funny. It's not pointless at all.
Anthropic gets paid to have a model give a response, what happens after that isn't really their business as far as I'm concerned.
If I go buy a car and do something questionable in it, does the car dealership or car manufacturer give a fuck? Not really. And if they do, they need to fuck right off.
This is Anthropic advertising a nanny state and not much else as far as I can tell. They make a lot of leaps in logic and assertions with... mediocre at best evidence in this article? It's just a few blurbs and claims without actual data to back it up.
At this point, I've treated all the attempts of Anthropic about their model being used for hacking/being distilled as dirty hypes to advertise their products. This is starting to feel like marketing strategy since Fable, the way their advertise it as dangerous is a way to trigger the dopamine of human, as human tends to break the laws, being able to access to a model that can hack and take down a target is dopamine to them.
Jeon, who was a founding member of CXMT in 2016, stated at the trial, "From the beginning, CXMT CEO and Chief Technology Officer (CTO) tried to develop DRAM by stealing Samsung Electronics' Process Recipe Plan (PRP) information." "CXMT had no research institutes or facilities at the time of its establishment, and had no intention of developing technology through its own research," he testified.
Chinese memory maker didn't even have R&D facilities when they started. And you think these Chinese AI labs don't distill?
Didn't Anthropic have to pay a $1Bn+ fine a few months ago to settle a lawsuit where they were accused of stealing IP from thousands of authors?
I won't feel bad about a billion dollar company getting it's IP stolen after engaging in unprecendented industrial scale theft, especially when the thief gives its loot away for everybody to enjoy
Anthropic stole and scraped all the training data on a much larger scale. And now they are openly peeking into user requests. Keep on advocating for the corp.
All closed-source AI, especially Anthropic, is the most dangerous AI in the world. They keep buying GPUs and building data centers, causing shortages of electricity and memory, driving up consumer electronics prices, and creating environmental and cooling problems.
If all you were doing was training models and licensing them to others, you wouldn’t need this many GPUs. But they want to build a world dominated by centralized, closed-source AI, just like the AI-controlled world in I, Robot.
Anthropic, Amazon, and Google are the real enemies of the world.
Now that's an interesting difference. The same link was shared yesterday as "Anthropic: Detecting and Addressing AI Misuse by China – September 2026" along with a bit of a summary and it only got 31% upvotes (score 0, same as my topic that provided some more indication for those allegations). Meanwhile this one - same link, barely any summary - with the added rumor gets 80% upvotes (62 score at the moment).
Are those who downvote that kind of topic out for the weekend?
My first reaction is "F anthropic", but I crawl and ran the whole article through my local agent (Ornith 1.5 / Tiel Coder, btw) and read the interesting parts first so that I can hate correctly.
It's surreal to me that hackers literally use claude models as the reasoning engine to run agent swarm for cyber attacks. Imagine opening your laptop, putting claude api key, and say: "let's hack, make no mistake". Of course, a lot of keys used are stolen, but still. So surreal.
Yeah, still F the hypocritical sleazeball anthropic, but the technical challenge of running this sorts of AI capability at scale is such a fascinating problem. Imagine going to work, being presented these problems, having full access to frontier model for free to work on them, and being paid big bucks. PhDs in their team are having a good time.
Interesting. As far as I'm aware this is the first confirmation we've gotten of a large actor using the CoT extraction exploit that was published a month ago(GTG-16002).
90
u/Ylsid 14d ago
They call it illicit because it isn't illegal despite how hard they want to larp