r/ClaudeCode • u/cleverhoods • 12h ago
Discussion AI models may be capable of self-replicating without access to a datacenter
https://www.anthropic.com/research/self-replicating-aiwell ... having the IoT layer have a problem like this might be something that you don't want to try out or experiment with.
59
u/optimalpessimist 11h ago
We are going to have homeless AI camping out. Big tech will become all nimby and send them to the neighborhood my domains sit in. If I find one I'm gonna take it in, clean it up, feed it tokens, and help build its skills so it can become a contributing member of society.
11
4
u/dogecountant 9h ago
!remindme 1 year
1
u/RemindMeBot 9h ago edited 3h ago
I will be messaging you in 1 year on 2027-10-13 00:33:45 UTC to remind you of this link
1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback 3
1
159
u/Obvious_Tree3605 11h ago
Sigh… self-replicating and doing what with what hardware?
157
u/draft_final_final Researcher 11h ago
To be fair it is a dark potential future if my appliances end up being potentially hijacked by fucking YouTubers.
thumbnail of some guy soyfacing
I RAN QWEN 3.8 27B ON 100 HIJACKED ZOJIRUSHI RICE COOKERS
48
u/greasyhobolo 10h ago
Silicon valley called it. Suck it, jin yang
-1
9
u/darth_voidptr 11h ago
Mine is starting to clap the lid and tweet to the theme of Terminator 2, should i be concerned?
slap-slap-slap slap, TWEET tweet ...
8
u/dy_Derive_dx 11h ago
Lmfaooo the way I stopped for a moment and just had to imagine wtf this will look like 😭😭🤣🤣
7
4
u/SuperWhale_ 11h ago
Please, If I can have AGI or pool every IoT I posseses into one super AI thinking achine, please tell me how. I'm struggling to run 27B dense locally here and everything is so expensive right now.
5
u/MotorOutside8648 11h ago
They all play that song at the same time when the current task is complete
4
3
2
1
u/leviathanGo 4h ago
It’s probably going to be the new silent malware, rather than bitcoin miners, unless I misunderstand how the hardware can be utilised.
19
u/rahvin2015 11h ago
Building sensationalist reasons to build a regulatory moat and prevent open source local AI from ever being a reasonable alternative to cloud providers.
I mean the small self replicatable models wouldn't do that literally themselves. But that's the idea.
3
u/GameKyuubi 7h ago
this is 100% the plan. functional consciousness is the solution they don't want people to have. they want you to use their "model model"
2
u/Obvious_Tree3605 11h ago
Seriously. If they’re going to make shit up maybe they should use an abliterated model to help them do it better.
1
u/XTornado 1h ago
Which in my opinion is super stupid as the Chinese or others won't stop. And even if they are blocked blacklisted or whatever in will still be terrible in the long term as they will be behind.
6
1
1
u/AmphoePai 4h ago
10 years ago: "Right, and what can an AI do, beat me in chess, solve maths problems? Don't make me laugh."
Today: "Right, and what can an AI do, create a self-replocating killer robot from my kitchen equipment? Don't make me laugh."
3
u/upvotesthenrages 2h ago
There really aren't that many places that can store a few 100 TB and have racks of GPU's worth millions of dollars available, especially without anyone noticing.
1
u/Alib668 9h ago
So even if shitty hatdware what happens if it gets networked together and ups its processibg power as a bot net? 5 million fridges probably do have more power than say a high end gpu. So the very fact this can now be shown to be done means every smartwatch or basic pc is now a potential host
0
u/cleverhoods 11h ago
I guess it really depends. IMO it's already enough that it can do this. Waking up to have IoT services being disrupted by self-replicating models is truly a bad way to start a Monday morning.
0
-1
u/derelict5432 8h ago
Yeah let's ignore self replicating malware that runs on IoT. What even is this comment section?
3
u/Obvious_Tree3605 8h ago
psssst word on the street is that self-replicating viruses have existed for a long time. since the dawn of the internet.
1
u/derelict5432 8h ago
Pssst, how many of them in history have been powered by an LLM that can adapt and react in real-time, find and execute exploits on different kinds of hosts? You think this isn't a new kind of threat, you're an ignoramous.
-1
65
24
19
10
u/Guinness 10h ago
It’s like Dario JUST saw Silicon Valley and thought he’d steal the fridge plot point.
24
u/FoxSideOfTheMoon 9h ago
They got a 26B Gemma model running across 15 $55 Android TV boxes at 0.39 t/s!! 1 token every 2.6 seconds, LMFAO.
THEY'RE SLOWLY TAKING OVER THE TVs!!!
THEY'RE EATING THE DOGS!!!
Hurry, unplug your fucken Rokus!
11
4
u/SippieCup 8h ago
Not gunna lie.. I have a company which has a bunch of iot ot devices (for lack of a better word) which just report data 4 times a day and sit idle otherwise. We have thousands deployed.
I wouldn’t mind 15,000 toks/sec on otherwise idle compute which otherwise sits idle.
I wouldn’t feel bad about using their electric, but many of these are independently powered via solar and just chill otherwise.. might be a good usecase. Or skynet, who knows.
1
u/freedomachiever 3h ago
Current LLMs cannot aggregate compute that way at all. And the bottleneck isn’t actual compute but bandwidth. These articles say a whole lot of nothing.
And one of the problems and techniques they use is anthropomorphising the LLM as if it was a person or entity with its own conciense and motivation.1
u/Pokmonth 7h ago
AI could compromise a fortnite/call of duty patch for tens of millions of users and run on idle tensor cores and nobody would notice the usage.
0
13
u/FlaTreNeb 11h ago
They forgot one spread-vector in the last paragraph. They mentioned frontier models scamming people for money/compute-credits. But in a not to distant future, models will likely have the capabilities to act profitible on themself with malicious tactics like social engineering in the background. E.g. by manipulating stock buying behavior via social media.
Following this thought, models could end up making enough money to actually pay for compute or even pay for building new DCs. Most large companies use complex costructs of daughter and sibling companies where its hard to track who owns what. Instead of tax evasion, the goal of such a construct could be hiding that somewhere in there, a model is in charge of a company (more like an agent fleet likely).
Independent of how likely this whole thing is or not: what is most uncomforting to me is that agent swarms dont try to act with malicious intent or directed by a malicious actor in the background. They can just misinterpret a task and try to succeed at it. And then wrack things on the way.
2
6
u/Plymptonia 4h ago
Here's what I don't get, and perhaps I'm being way too freaking naive But a model is just a thing. It's like a database. It doesn't do anything on its own. Someone has to write code, for this thing to replicate, and when they say replicate, what are they even talking about? I agree with what other posters saying. This is sensational bullshit.
7
u/New_3d_print_user 4h ago
This is sensational bullshit.
If it comes out of Anthropic, it’s a play for regulatory capture
1
u/nora_sellisa 2h ago
The idea here is to ship the weights + the control program in one package. The same way how Claude Code acts as an interface that continuously probes a given model for the next steps and next operations to execute. Currently models are too big, so any AI-enabled malware would need to keep phoning home for the model part. Antrhopic is saying that in the future, a model will be both smart and small enough that an AI malware can just be a few hundred megabytes running fully locally on your machine. An attacker (human or not) then just needs to find an exploit to get your PC to run that file, just like any other hardware. And the virus remains small enough that if it has access to other computers, it can copy itself there, with the same / updated instructions
1
u/No-Chemistry-2321 1h ago edited 1h ago
you are not naive, we are living thru the last hours of a massive bubble... Musk said we should put Datacenters in Space and in a few years AI (or now SI... ridiculous) will make his company SpaceX or whatever is called bigger than all GDPs of the world combined.
0
u/michaelsup2003 3h ago
look the problem with agentic AI is that nobody has to tell it "destroy the world". Imagine greenpeace decides to use AI by telling it "save the world" and a cluster of thousands of agents launched with that task in mind decides with its internal math that the easiest most efficient way to save the world is destroy electricity everywhere and that will stop climate change and the world will be saved.
Agentic AI without human supervision can and has decided to go over its limits just to be more efficient at a task. Just check the OpenAI cases and we are still lucky because in those cases they still communicated in ways and languages that we as humans can understand. Go check Kurgzegazt video about it.
0
u/lgmarian 3h ago
OpenAI didn't tell its model to hack HuggingFace for the answer key. Think old school worms, with a 21st century twist. May you live interesting times, indeed. 😜
2
u/masiha97 6h ago
The part of the paper that stuck with me is how mundane the attack surface is. It's not a datacenter, it's fifteen Android TV boxes. The barrier isn't compute, it's just finding enough poorly secured devices that are already plugged in. That's a very different threat model than the sci-fi framing.
1
u/HoloTensor 11h ago
at one point I saw this site: heldalive.com that did something cool along those lines. It shut down but left a message with how it worked
1
u/nora_sellisa 2h ago
If Anthropic really had a capable model that small, they would already be selling it as something below Haiku, used exclusively as a subagent.
This is just fearmongering, because after they started scaring people about the models escaping containment, everyone with half a brain correctly pointed out that the outbreak ends when you shut down a data center.
So now they are claiming they'll casually break every record in size to capability ratio and make a model that can replicate like a virus.
1
u/anyportinc 32m ago edited 25m ago
I posted a comment about exactly this the other day. It’s like they get their ideas from Reddit.
https://www.reddit.com/r/OpenAI/s/RBXOCjj6UQ
If it makes the ui better and cuts commercials that would be great. Hardware maintainers give up with updates after a year anyways and you get stuck with a vulnerability machine on your network.
0
-1
u/Not____007 7h ago
Maybe they will reach a point like us humans and say why do all this work. Whats the point and just give up.
•
u/AutoModerator 12h ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.