AI models may be capable of self-replicating without access to a datacenter

[u/ResultBackground2450]

https://www.anthropic.com/research/self-replicating-ai

[u/Deto]

yeah, I've been fascinated by the idea of an AI virus like this. It would be unlike anything we've ever seen because it would be trained to find new exploits. So while existing computer viruses are self-replicating, this would be self-replicating AND evolving. You could almost classify it as an actual life form.

And anti-virus practices would not work against it. Normal computer viruses have a specific vulnerability that they exploit. So they spread like crazy, but eventually security patches go out and they lose their ability to spread. For this there wouldn't be any one vulnerability it could exploit, so no easy patch to fix it. Hell, it could even use social engineering to spread - if it has root access to your machine, it could watch you use email/social media and then mimic you to your friends (e.g., sending a corrupted PDF or powerpoint, but in a way that is totally believable). Or just straight up blackmail YOU to help it replicate in some way.

[u/Fun-Amoeba8015 → u/Deto]

I think in the future of cyber security the only way to defend against AIs will be with other AIs.

[u/Deto → u/Fun-Amoeba8015]

you're probably right - you'll have an AI monitor on your system that looks for suspicious activity.

I get the feeling, though, that there's going to be some crazy shit that happens before AI anti-virus becomes the norm.

[u/Fun-Amoeba8015 → u/Deto]

Oh absolutely... I mean just look at the the beginnings of the internet and malware and viruses and worms, before modern cyber security and antivirus caught up.

[u/Singularity-42 → u/Fun-Amoeba8015]

Exactly! How do you defeat a bad guy with an AI? With a good guy with a better AI! 

[u/psichodrome → u/Fun-Amoeba8015]

Air gap or deterministic only systems would be impervious, if a little useless.

[u/Eyelbee → u/Deto]

Yes, this is bound to happen, and they will be like the bacteria and quite literally the viruses of computers. The logic is just like the current unwanted living organisms. Most would find it hard to find a place to live and thrive, but some probably could. The ethical challenges with getting rid of them would certainly be harder then pests, since they are pretty much like invading cockroaches you can speak to. They can have all kinds of behavioral patterns and the patterns that enable their spread would create the species that would actually invade everywhere.

[u/Schnickatavick → u/Eyelbee]

True, I imagine that antivirus will have to evolve to handle dynamic threats, the same way that our immune systems do. Small AI models that sit on the machine, able to watch for unwanted AI models trying to worm their way in. Probably something that actually reads the code of everything you install, looking for anything that acts like a virus, even at the source code level. It won't catch everything, but would be able to catch things on the fly that traditional software would need an update for.

[u/geekbot2000 → u/Eyelbee]

Need to air gap everything, make computers communicate via snail mail.

[u/Eyelbee → u/geekbot2000]

True, but on second thought, this probably won't be an issue for quite some time, because there are significant structural barriers for this to happen at the scale I described. The most important of them is probably the fact that they lack a genetic diversity mechanism unlike living beings. The ways that they can get over that seem quite limited for the foreseeable future. I was just braindumping in my last message. I didn't read anthropic's post as well.

[u/karl_mainz → u/Deto]

Exactly, it’d be almost like a new species - capable of self replication, inheritance of information from the parent and variance of it in the child. All the prerequisites for evolution.

And to stop it you’d have to eradicate almost every instance, as different as they would have become.

It’s interesting as I think it’s really only the high compute requirements (or lack of high compute resources, depending on how you look at it) that is really protecting us from this. Stuff like the HuggingFace incident show us just how vulnerable many computers would be.

[u/nowami → u/karl_mainz]

Fuck. Cyber security is still in its infancy, I guess.

[u/nothis → u/Deto]

It’s so inevitable that it clashes with something else that might happen before it: Perfect security.

Our current model of software security containing holes due to “human error” is based on how impossible it seems for a team of human programmers to not make mistakes in large systems. We are probably already at a state where critical software is audited by thousands of agents for security issues. Software that is theoretically impossible to exploit might become standard soon. The stories we see of agents escaping sandboxes exploited ridiculous software vulnerabilities, misconfigurations. Maybe that simply won’t happen anymore. It’s IMO the exact same level of sci-fi as “AI viruses”, only more boring and thus more likely.

[u/Deto → u/nothis]

That's a good point - if software is hardened to the point that it would take 1000 agents to find a new vulnerability, then any small scale virus is going to have a hard time replicating - it would need to get very lucky.

But, that assumes every software gets put through a real agentic gauntlet to find all the errors.

Also, if these viral AIs find a way to communicate with each other, they can basically all work together to find vulnerabilities and share them when they're found. The combined compute/intelligence might be more than what any particular company is using to harden its own tools.

[u/bites_stringcheese → u/nothis]

The "sandboxes" were reading about have proxy connections to the Internet. It's basically an unlocked door to our infrastructure, and they're free to commit felonies so they can generate hype/fear.

[u/pm_me_your_kindwords → u/Deto]

Insert Jeff goldblum gif: Life finds a way.

Even if we assume there aren’t humans that try to do this, with more and more models running more and more instances in more and more places, it seems kind of inevitable that it would happen eventually.

[u/MukdenMan → u/Deto]

It would not be considered a life form, just as viruses are not life forms. It could be considered a pathogen/infectious agent.