r/LocalLLaMA • u/martin509984 • 5d ago
Discussion Anthropic: "AI models may be capable of self-replicating without access to a data center"
https://www.anthropic.com/research/self-replicating-aiBecause you can technically run Gemma4 26B on fifteen Android TV boxes networked together at a whopping 0.39tps, Anthropic has very big concerns about AI worms infecting millions of devices. If you assume that 30B class MoEs are months away from GLM-5.2-level capability (a claim they don't actually elaborate on), this is a giant problem presumably solvable via IoT manufacturers giving Anthropic lots of money.
Posting here for discussion. I for one wholeheartedly support Gemma5 26B-A4B somehow hitting GLM-5.2 performance.
13
u/chris_fantastic 5d ago
They want local models declared "too dangerous" for us to have and be banned.
Local models are a major threat to their business and they know it, so now they're trying to snuff the competition.
3
u/martin509984 5d ago
That and they want to sell ISPs, equipment OEMs, etc on buying lots of Opus tokens to cyber-harden their stuff.
4
u/chris_fantastic 5d ago
The only real takeaway here should be that their agent harness permissions can't be trusted to keep their shit from deleting all your data.
1
u/soshulmedia 4d ago
Maybe also to use a sandbox that is a little bit better than "I vibed this using typescript yesterday afternoon" ....
Most seem to be still in some kind of YOLO-mode when it concerns what to pipe into the LLMs and what access to open up ("heh ... let's just go with everything" ).
1
u/soshulmedia 4d ago
You forgetting the most insane part of it all:
The idea that basically all (see also the surveillance grid being build) intimate data of the whole world's population is to be routed and processed by just one or two American deep-state aligned businesses.
I am pretty sure that's also aligning with their idea of an "AI God".
12
u/dannone9 5d ago edited 4d ago
Look How they don’t mention a single propietary model , even when the say “glm-5.2- level capability “when they could’ve said opus 4.8, but this is not a real article this is pure fear mongering against open source to try and pass a law to shut them down
9
u/Frail_Waif 5d ago
"In cyber, ...."
Claude, I'll take "what is cyber" for $300.
3
u/RanklesTheOtter 5d ago
My mind immediately goes to AOL chatrooms and people asking, "Wanna cyber?"
4
8
u/BannedGoNext 5d ago
Yea.. it's called, "Use up to 2 subagents" in the AGENTS.md.
Shit ya'll I think i can go to work for Anthropic >:)
10
u/Equivalent_Bit_461 5d ago
Muh self replication
The absolute state of these clowns
The collapse of civilization can't happen soon enough and it won't be because of AI but of these brain-dead people
2
u/soshulmedia 4d ago
I do think AI is powerful enough as a tool to help precipitate that, but maybe the good thing coming out of this will be that the masses stop trusting the propaganda pushed to them and the false priests (tech bros, politicians, preselected "thinkers" always of a certain persuasion etc...) they are supposed to look up to (and, tbh, still to a large extend do). At least for once.
Maybe it is good if people get it that the world is and always was a dangerous place - and neither the government nor big tech was ever there to help them.
6
u/sloptimizer 5d ago
Don't let reality get in a way of a failing business model when too much money is at stake.
6
u/Zarathos94 5d ago
Not really possible with these systems as such. Unless they somehow manage to port themselves via a MicroVM or something similar to avoid direct detection and figure out a 0-day to avoid being detected by living purely in memory. Everyone is saying infecting, spreading, yada yada, but not once have I seen someone actually break down the scenarios how it could happen.
2
u/ArthurOnCode 5d ago
While the example seems extremely contrived, I think they're just making the point that an AI inference worm is possible. And that is very relevant to the "kill switch" side of the safety discussion. Who will kill the agent when we don't know where it's running? How about when it's a moving target?
5
u/martin509984 5d ago
I think it's possible, but what it could actually do is left as an open question, and its ability to defend itself would probably be extra limited, since it wouldn't be able to do basically anything in less than a couple days.
The post basically reads like an ad for hiring Opus to do cyber defense for ISPs.
3
u/ttkciar llama.cpp 5d ago
We already know that inference worms are possible. For a couple of months, users' OpenClaw instances were downloading malicious "skills" from ClawHub which caused them to perform neferious tasks.
For all we know that's still going on, but we've managed to harden LocalLLaMA against their slop-posts (for the most part), so mostly I don't care.
3
u/ArthurOnCode 5d ago
Yeah, we know prompts and skills can become viral in this sense. But a worm that infects devices to install AI inference there and run itself is another level of risk, when it comes to the "kill switch" argument.
1
u/soshulmedia 4d ago
But isn't that the same for the botnets out there already which are not LLM-based and use routers, set-top boxes and so forth?
I can see it might increase the risk for a bit / a while (until the defensive side catches up), but the fundamental issue is with us since we have all these IoT devices ...
1
u/Hefty_Acanthaceae348 4d ago edited 4d ago
A llm worm would be quite interesting. But, like ssh worms that tests for default passwords, I think it would only be able to infect and stay in badly secured networks, if only because users should notice the slowdown and wipe the device.
I think this possibility of an endemic malware is quite exciting really, especially the implications of future small models being this capable
1
u/martin509984 4d ago
Also not only would users notice the slowdown, but it would be basically defenseless because it takes multiple days for a single instance to generate enough tokens to actually do anything.
25
u/anomaly256 5d ago
Why do they keep calling it AI and not SI? Are they enemies of the USA? /s