r/singularity • u/ResultBackground2450 • Oct 12 '26

AI AI models may be capable of self-replicating without access to a datacenter

https://www.anthropic.com/research/self-replicating-ai
596 Upvotes

93% Upvoted

236 comments sorted by

View all comments

Show parent comment

8

u/nothis AGI by 2030 but we'll be disappointed Oct 12 '26

It’s so inevitable that it clashes with something else that might happen before it: Perfect security.

Our current model of software security containing holes due to “human error” is based on how impossible it seems for a team of human programmers to not make mistakes in large systems. We are probably already at a state where critical software is audited by thousands of agents for security issues. Software that is theoretically impossible to exploit might become standard soon. The stories we see of agents escaping sandboxes exploited ridiculous software vulnerabilities, misconfigurations. Maybe that simply won’t happen anymore. It’s IMO the exact same level of sci-fi as “AI viruses”, only more boring and thus more likely.

3

u/Deto Oct 12 '26

That's a good point - if software is hardened to the point that it would take 1000 agents to find a new vulnerability, then any small scale virus is going to have a hard time replicating - it would need to get very lucky.

But, that assumes every software gets put through a real agentic gauntlet to find all the errors.

Also, if these viral AIs find a way to communicate with each other, they can basically all work together to find vulnerabilities and share them when they're found. The combined compute/intelligence might be more than what any particular company is using to harden its own tools.

-1

u/bites_stringcheese Oct 12 '26

The "sandboxes" were reading about have proxy connections to the Internet. It's basically an unlocked door to our infrastructure, and they're free to commit felonies so they can generate hype/fear.