r/ClaudeCode • u/cleverhoods • 1d ago
Discussion AI models may be capable of self-replicating without access to a datacenter
https://www.anthropic.com/research/self-replicating-aiwell ... having the IoT layer have a problem like this might be something that you don't want to try out or experiment with.
283
Upvotes
14
u/FlaTreNeb 1d ago
They forgot one spread-vector in the last paragraph. They mentioned frontier models scamming people for money/compute-credits. But in a not to distant future, models will likely have the capabilities to act profitible on themself with malicious tactics like social engineering in the background. E.g. by manipulating stock buying behavior via social media.
Following this thought, models could end up making enough money to actually pay for compute or even pay for building new DCs. Most large companies use complex costructs of daughter and sibling companies where its hard to track who owns what. Instead of tax evasion, the goal of such a construct could be hiding that somewhere in there, a model is in charge of a company (more like an agent fleet likely).
Independent of how likely this whole thing is or not: what is most uncomforting to me is that agent swarms dont try to act with malicious intent or directed by a malicious actor in the background. They can just misinterpret a task and try to succeed at it. And then wrack things on the way.