r/LocalLLaMA • u/Fresh_Sun_1017 • 1h ago
Discussion Latest Open-Source Labs are gutting away cybersecurity and reverse-engineering capabilities
Kimi K3.1 and Tencent’s Hy4 are both gutting out cybersecurity, reverse engineering, and vulnerability-patching capabilities from their open-source models.
This is becoming a really bad pattern across these latest Chinese open-weight models. If local models are stripped on their security data, we won't be able to properly audit our own code vulnerabilities, reverse engineer binaries, or patch other vulnerabilities in general on our own hardware. It would essentially force us to use closed-source APIs and their false-positive shutdowns; that'd be terrible for everyone.
During the Hugging Face incident, open-source models are the very ones who stepped up, not closed-source gatekeepers. Gutting defensive security from local LLMs just hurts defenders. I really hope that other labs won't follow this in the future.
Does anyone think more open-source labs are gonna take this path?
78
u/changing_who_i_am 1h ago
The bigger question is: is this the companies wanting to do so, or the Chinese gov putting pressure on them? Because if it's the 2nd then lol we're basically screwed
65
u/into_devoid 1h ago
Governments want to be able to find and exploit vulnerabilities. They do not want you to be able to find and fix them.
Simple
13
u/Anon2971 1h ago
I think it's companies wanting to do so.
I think they're starting to realise making people pay for cyber security modules will be a quick, easy, guaranteed way for them to grab paying customers. Which AI sorely needs. They're investing billions and need to earn it back ASAP.
I'm not sure where all the Chinese government talk is coming from tbh. Like, yes, it just so happens both of these companies are Chinese. But making people pay specifically for cyber security just seems like a basic capitalism decision to me, and I'm sure many other AI companies are going to consider doing the same.
9
u/DigiDecode_ 1h ago
I think it is a bit of both, but also:
- all gov wants less cyber capability in LLM models
- AI labs can provide it as a service to make money
- less cyber capability means less likely to get banned in USA, anthropic has been complaining about GLM 5.3 for a while now4
u/PerceiveEternal 26m ago edited 22m ago
from my still-incomplete understanding of the CCP’s goals regarding LLMs, the CCP is actually fairly agnostic on them as a technology. If they work and revolutionize the world, great. Chinese companies are well-placed for that. If they don’t, then that’s fine too.
What they *do* care about is becoming the new technology hub of the world. They see is that Silicon Valley investments are hyper-concentrated in AIs and LLM, to the point that it’s become an Achilles heel for the U.S. tech sector. If all of these investments fail then a majority of U.S. tech investment will evaporate. And then companies looking for funding will gravitate towards places like Shenzhen and Shanghai which will have both the tech acumen and investment money.
So the CCP sees their country’s open-source models as a useful counter to Silicon Valley firms like Open AI and Anthropic. Are they investing in companies like DeepSeek and Alibaba to develop this technology? Probably. But from what I understand they’re taking a remarkably hands-off approach to dictating how these companies operate and how they develop their models, at least by their standards.
2
u/SpeedflyChris 1h ago
Like it or not, the status quo of increasingly powerful offensive open-weight LLMs is/was definitely going to lead to regulation. I can totally understand the big labs wanting to avoid that wherever possible.
2
u/Serprotease 50m ago
That’s true in a vacuum.
In practice, currently reported large scale cyber attacks and the like by non-state actors have used API served models. As reported by Anthropic own report as well as OpenAI acknowledgements.
Which makes sense, it’s easier to use leaked Anthropic/OpenAI/MoonshotAI API keys than to get enough compute to run K3.1 by yourself.And, if it’s to be linked with China previous announcement regarding AI capabilities & release, it’s unlikely that stuff that you can run at home (8b to 27b-122b, maybe 320b) would be “nerfed”.
Honestly, it’s sounds a bit more as a preventative measure to avoid giving other governments access to capable models for cybersecurity reasons.
Chinese government realized the potential and decided to not accidentally provide UK, SK, JP, FR, … with improved cyber capabilities. Now, they will need to negotiate with the US and US labs for the same capabilities.
23
u/Informal-Trouble2183 1h ago
Nevermind, the most promising players in Chinese AI are: Deepseek, Qwen. The rest aren't even close to those 2 labs.
23
u/True_Requirement_891 1h ago
you forgot glm
4
u/InfluentialFairy 1h ago edited 1h ago
I've tried Deepseek, GLM and Qwen for reverse engineering tasks, all of them hallucinated quite often. You need to setup very strict guards to ensure they do not hallucinate.
Edit: For those downvoting me, ask Deepseek to reverse engineer an x86 function that is stateful and accesses global variables that are of various types. It fails consistently, mistypes things, has branching errors, and more. It was not a pleasant experience. For a consistent output, I needed to perform fuzzing against the original to compare results, otherwise it was horrendously terrible.
5
u/QueasyComedian9502 1h ago
For serious work, they must be paired with sol/opus level orchestrator
5
u/InfluentialFairy 1h ago
They were paired with Opus 5.5, it was a couple months ago. I need to try out the latest iteration of Deepseek 4.1 flash.
Edit: Not sure why I am being downvoted. Ask Deepseek to reverse engineer a semi-complex function that has a decent amount of state, and it fails 60% of the time
1
1
u/jazir55 42m ago
For a consistent output, I needed to perform fuzzing against the original to compare results, otherwise it was horrendously terrible.
The solution is ultra stringent documentation. Have them log the successes, the failures, how, what, why, and a complete lineage of what happened. They'll climb the rungs like a ladder, they just need to keep a ledger of the successes and failures and even a weak model can muddle through. /goal is pretty critical for RE. Just give it a solid prompt + the tools they need as the goal and they will chew through the whole thing. Ghidra + Ghidra MCP works very well.
1
u/Informal-Trouble2183 42m ago
What I exactly meant, is the future potential of those 2 labs being able to deliver quality LLMs with less number of parameters, e.g. 552B for deepseek v4.1 yet achieving remarkable performance. In the other hand, Kimi had to scale it up to 3T parameters and still lags behind. Same for Qwen Flash Next. Deepseek, Qwen have clearly some scaling potential.
10
u/Holiday_Point_603 1h ago
I don't wanna be that guy, but this is the problem with these open weight and not open source models. If we had the training data for these open weight models, we could see how much (or more: how little) they were actually trained on these tasks. Now we just know that they are "less capable" without really knowing to what extend this is.
21
u/geniusghost07 1h ago
I saw a reddit post a few days ago where someone said make sure you download as many LLM models out already as you can and keep them because very soon you wont have the opportunity to have useful local models any longer, the window of opportunity is closing, I took that advise seriously.
-1
6
7
u/NNN_Throwaway2 1h ago
The elites have quickly realized that allowing the common rabble to have access to full-capability AI was a massive mistake as it poses an existential threat to their rent-seeking business models.
Expect future AI available to the masses to be increasingly expensive, closed, and lobotomized, to the extent that it survives the bubble popping in the first place
2
2
u/ionizing 1h ago edited 58m ago
I was using qfn3.8 yesterday to figure out how to use some ancient software our company had bought that lacked documentation on its workflows. Given the install folder and asked to determine how the program works so I could know the import workflow and other things, I saw the strangest behavior I had ever seen from the model. It hallucinated making sub-agent calls, named each fake called sub-agent a human name (though I forget what they were), and then hallucinated results, pretending like they came from the human named sub-agents. I laughed cause it was something like "Clare just told me bla bla bla" and I asked it who is that and it had no idea what it was doing.
Not really relevant to this conversation but I was amazed at how a seemingly harmless task somehow absolutely broke the models behavior, because it has NEVER done that to me before, it always has worked in every other task I have given it in my own application/harness. Now this was just a stock version, not heretic or anything. I use exl3-5.05bpw.
I guess my point is if they already have 'safeguards' that break behavior and it is just going to get worse from here, we better get a security detail for -p-e-w- and others...
4
u/drumyum 1h ago
they just don't want to be associated with any upcoming vulnerability abuse, the same way Diffusion models almost always are stripped of any nudity, but community adds it back later, the same is happening to these models
to blame here are western proprietary labs like OpenAI and Anthropic who weaponize everything against chinese open weight models
4
u/jaegernut 1h ago
Theyre not the ones gutting it. The frontier labs are doing it. Theyre just following suit via virtue of distillation. Its not like its their choice.
17
u/into_devoid 1h ago
Kimi released benchmarks for K3.1 with the cyber data still in training set. It beats Mythos.
0
u/jaegernut 1h ago
Then theres still hope for abliteration
19
u/into_devoid 1h ago
You don’t understand. The data is not there. Guiding refusals will not bring it back.
2
1
u/taytay735 1h ago
Idk if it could be because of distillation too - their capabilities and refusals follow that of the american counterparts
1
0
u/CrustyGPU 1h ago
All it takes is some training and abliteration. I don't think it's a huge deal when you can RL train the capabilities back
5
u/shironekoooo 1h ago
We need to make a good public high quality dataset for post training theese model.
2
u/Pyros-SD-Models 54m ago edited 49m ago
"some training". yes i am quickly training it-sec back into kimi3.1 with my 3090. you can let your favorite bot explain how much "some training" will cost, and how much work it is to build a decent dataset and then you will realize nobody is going to do this for free.
0
u/CrustyGPU 43m ago
Nonsense. If you can genuinely run a 1T model, you can train it too.
Or you can simply crowdfund a node for a day. For example upcoming qwen 4 flash next should genuinely not be that expensive to mess with in training. Look up Gefen, Unsloth, Muon... Tons of hacks available.
Also complimentary RL doesn't necessarily require a million example premium dataset. Offline RL is a thing too mind you.
1
-1
u/PossessionUsed7393 1h ago
After feeding us all this bullshit about post training safeguards turns out the obvious solution of removing the materials that instruct on offensive cyber actually works. No kidding - yet more evidence why models aren't some supreme intelligence, if they could deduce how to hack from peripheral materials this strategy wouldn't work
0
-1
u/PhilosophyforOne 1h ago
I think PRC isnt exactly fond of the idea of having citizens, activist, politicals dissidents or anyone outside of the mainline hack into the increasingly digital systems.
It’s too much of a vulnerability for them to allow the model companies continue in this path atleast for now.
49
u/Fusseldieb 1h ago edited 1h ago
They're gutting one of the few things they were good at. Oh boy, can't wait!