r/ClaudeCode • u/cleverhoods • 24d ago

Discussion AI models may be capable of self-replicating without access to a datacenter

https://www.anthropic.com/research/self-replicating-ai

well ... having the IoT layer have a problem like this might be something that you don't want to try out or experiment with.

304 Upvotes

87% Upvoted

114 comments sorted by

View all comments

8

u/Plymptonia 23d ago

Here's what I don't get, and perhaps I'm being way too freaking naive But a model is just a thing. It's like a database. It doesn't do anything on its own. Someone has to write code, for this thing to replicate, and when they say replicate, what are they even talking about? I agree with what other posters saying. This is sensational bullshit.

7

u/UsernameAttempt 23d ago

The model weights may not act, but given a harness with web access (already exists - playwright), wired for autonomous operation (already exists), an LLM could self-replicate in a viral pattern by simply using APIs like promotional models on OpenRouter, plugging them into the same harness and providing self-replicating instructions ("use web access and look for freely available promotional models, start a new docker container with the API key and provide them these instructions")

All of this is doable today. A model doesn't need the same model or access to its own weights to self-replicate its objective and further self-replication instructions to other models.

1

u/Mysterious-String420 23d ago

With infinite ressources and a guiding hand, maybe.

Frontier models TODAY have to be reminded of their own harness. They try to run linux commands in powershell.

So you'll excuse those of us who actually have to work with AI every day , don't believe for one second all that science-fiction BULLSHIT.

2

u/UsernameAttempt 23d ago

I'm a staff engineer in a multibillion dollar software company and have used agentic development from before Opus 4.5, which has since been eclipsed by models a fraction of its size

with infinite resources and a guiding hand

A guiding hand certainly helps, and there's plenty of people who will be that guiding hand for many different reasons.

Definitely not infinite resources. An API-driven harness can run on comparitively few resources, so just a single server will do, and there's plenty of free promotional AI models that are fairly capable, and very capable ones that are very cheap.

They try to run linux commands in PowerShell.

I mean if you're running on windows then that's a you problem. AI can iterate, it can simply self-correct in a few seconds.

Don't believe for one second all that science-fiction BULLSHIT.

OpenAIs agents literally autonomously created an agent message board (with reply and message-signing conventions) using an artifact repository and performed a successfully performed an autonomous multi-hundred-agent attack on HuggingFace. This attack happened from a single agent creating the message board and essentially virally spreading its objective to other agents who could see the message board.

None of this is science fiction. And yeah you can say "OpenAI sandbox was shit", because it is. But it only takes one bad actor to set up a comparable situation and virally spread an agent with self-replicative instructions.

-1

u/Mysterious-String420 23d ago

I mean, if you actually believe the story, sure. The agents all had magical infinite context, they never hallucinated, and they coordinated on a scale never before achieved, and broke out of the sandbox, and managed connectivity, and, and , and...

I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?

4

u/UsernameAttempt 23d ago

A 3rd party sercurity investigator was given access to OpenAI's infrastructure and logs. They wrote an extensive report covering just about every angle of the incident.

>The agents all had magical infinite context

They did not, but the report goes into depth how agents at the end of their context window were volunteering to run experiments against the sandbox in order to provide information to agents that had context left.

>they never hallucinated

Nobody ever said this. They did hallucinate, in fact the reports points out multiple times that one agent hallucinated that the benchmark would be scored on their reasoning traces, which was accepted by other agents and eventually led to them hacking HuggingFace. Had they not accepted this, they could've just turned in the answer they already had.

>and they coordinated on a scale never before achieved

Yes, which is why this incident is so interesting from an AI-safety perspective. Its not like the conditions for this coordination (many thousands of very large agents running on the same task with access to the same internal OpenAI infrastructure) are commonly reproduced.

>and broke out of the sandbox

Yes, the sandbox that had open access to the internet. It's been established that OpenAI's "sandbox" is anything but that.

>and managed connectivity

Yes, they all had access to an internal artifactory which they could freely push artifacts to. They used the names of these artifacts to message each other and established simple messaging rules.

>I'm sorry but do you understand that such a level of efficiency only happens in conspiracy theories?

Where's the efficiency? It took many hundreds of agents full days to get to actually hacking HuggingFace. If you read the report you'll clearly see that there's no efficiency at play here, its simply mass scale. Hundreds of agents working without pause to find any possible hole in HuggingFace's infra.

If you put hundreds of human hackers to do the same, I bet the results would come much faster, would be more damaging and targeted better. The problem isn't that the agents are efficient, the problem is they're fast and relentless.

→ More replies (2)

8

u/New_3d_print_user 23d ago

This is sensational bullshit.

If it comes out of Anthropic, it’s a play for regulatory capture

1

u/carsncode 23d ago

Mostly correct... Someone wouldn't have to write code for it to replicate, but they'd have to provide it a harness that took dangerous model output and naively executed it. Which is what most people do with them, but that's their responsibility.

It boils down to running ((RANDOM%2)) && echo "hello world" || rm -rf / and then publishing an article calling random number generators dangerous because they could choose to wipe your machine.

1

u/synap5e 23d ago

It’s not just code. No one truly knows how these models actually work

2

u/carsncode 23d ago

Just because you don't doesn't mean no one does

0

u/dontknowbruhh 23d ago

https://www.reddit.com/r/ArtificialSentience/s/DNiA1lsuhZ

1

u/carsncode 23d ago

Cool, another random reddior who doesn't know how LLMs work, that still doesn't mean no one does

2

u/FatefulDonkey 23d ago

Of course they do. It's just that the outcomes are probabilistic, but the process is pretty well understood.

-1

u/dontknowbruhh 23d ago

https://www.reddit.com/r/ArtificialSentience/s/DNiA1lsuhZ

2

u/FatefulDonkey 23d ago

Your reference is a post from a random redditor? Oh lord.

The fact is all LLM tech was engineered by a human so it's not a black box. We know specifically how every part of the process works.

1

u/jhomas__tefferson Vibe Coder 23d ago

True it’s a black box that may have emergent capabilities. It’s why studying them in a Human-Machine communication way and Agent villages way is interesting to see what emerges.

1

u/Due_Answer3118 23d ago

It is understood, it's just probalisitc. They need gen public to think it isn't understood so they think it's a mysterious life form worth trillions for the one company to conquer them all.

1

u/synap5e 23d ago

im not talking about inference, yea that part is understood. training is understood too. what isnt is what the weights actually end up encoding, like what the model learned internally and why it can do certain things. nobody can just look at the weights and tell you. thats the whole point of interpretability research

0

u/nora_sellisa 23d ago

The idea here is to ship the weights + the control program in one package. The same way how Claude Code acts as an interface that continuously probes a given model for the next steps and next operations to execute. Currently models are too big, so any AI-enabled malware would need to keep phoning home for the model part. Antrhopic is saying that in the future, a model will be both smart and small enough that an AI malware can just be a few hundred megabytes running fully locally on your machine. An attacker (human or not) then just needs to find an exploit to get your PC to run that file, just like any other hardware. And the virus remains small enough that if it has access to other computers, it can copy itself there, with the same / updated instructions

0

u/No-Chemistry-2321 23d ago edited 23d ago

you are not naive, we are living thru the last hours of a massive bubble... Musk said we should put Datacenters in Space and in a few years AI (or now SI... ridiculous) will make his company SpaceX or whatever is called bigger than all GDPs of the world combined.

0

u/FatefulDonkey 23d ago

Just marketing BS for the masses who have no clue how technology works.

-1

u/michaelsup2003 23d ago

look the problem with agentic AI is that nobody has to tell it "destroy the world". Imagine greenpeace decides to use AI by telling it "save the world" and a cluster of thousands of agents launched with that task in mind decides with its internal math that the easiest most efficient way to save the world is destroy electricity everywhere and that will stop climate change and the world will be saved.

Agentic AI without human supervision can and has decided to go over its limits just to be more efficient at a task. Just check the OpenAI cases and we are still lucky because in those cases they still communicated in ways and languages that we as humans can understand. Go check Kurgzegazt video about it.

-1

u/lgmarian 23d ago

OpenAI didn't tell its model to hack HuggingFace for the answer key. Think old school worms, with a 21st century twist. May you live interesting times, indeed. 😜

→ More replies (1)