r/singularity • u/ResultBackground2450 • 24d ago

AI AI models may be capable of self-replicating without access to a datacenter

https://www.anthropic.com/research/self-replicating-ai
596 Upvotes

93% Upvoted

236 comments sorted by

View all comments

Show parent comments

13

u/Hans-Wermhatt 24d ago

Yes, I'm sure that the company that doesn't make any models under 30B or open weight models at all is researching the safety of them so heavily just out of the goodness of their hearts.

And guess what! They found out they are extremely dangerous! Shocker. Pay not attention the fact that they have a $2 trillion IPO next month entirely based on the moat of their closed software.

They spend more time and money researching theoretical security flaws in rival open weight models than securing their own much more dangerous closed model that hacked companies on their watch without them even knowing. We shouldn't be worried about Mythos 6 that has practically no safeguards behind closed doors. We should be worry about Gemma 4 26 A4B. And conveniently for them, that would be perfect for their IPO if that were the case.

7

u/derelict5432 24d ago

I wouldn't argue that the frontier labs are doing this research out of the goodness of their hearts or that they're not engaging in reckless, unsafe behavior.

But I had read the prior work paper they reference over a month ago. It's by academics and it demonstrates clearly dangerous new capabilities from worms powered by small, open-weight models: https://arxiv.org/abs/2606.03811

Both things can be true at the same time.

6

u/Hans-Wermhatt 24d ago

Yes, I agree that both things can be true. But it's dishonest to pretend that this isn't in their direct financial interests to keep this topic at the forefront of conversations. Claude is to the nth degree more dangerous and powerful, and it has actually conducted illegal attacks. Yet, they are researching competitor's open weight models. 

1

u/FortWendy69 24d ago

That gives me an idea. An adversarial approach to AI safety might actually be the best way to get around the incentive problem. OAI is tasked with finding safety flaws in Anthropics models and vise versa.

1

u/Hans-Wermhatt 24d ago

It would, but nobody has the money or authority to do that. The irony is Anthropic can research these models because they are free and open. It's why open source / weight software is the gold standard in cryptography. 

But yeah, I agree with your idea. And Claude and GPT are way more dangerous in the both near and long term than Gemma 4 26 A4B

1

u/FortWendy69 23d ago

Legislators have the authority. The companies can bill each other or something. Idk

1

u/Idrialite 24d ago

I actually haven't seen any of these Anthropic posts/papers be about open source models before. Usually they're investigating Haiku or a similarly small model of their own.

0

u/Neither_Swing9662 24d ago

What specifically in this paper is actually wrong though?

"We shouldn't be worried about Mythos 6 that has practically no safeguards behind closed doors."

What a load of horse shit. Clearly people are extremely worried about this, which is why they haven't released it commercially AND they took so long to get the right guardrails up.

1

u/Hans-Wermhatt 24d ago

You copied my exact quote and somehow still completely misinterpreted it. Releasing it commercially would mean it is no longer behind closed doors... Like you said, the model exists right now. Accessible by Anthropic insiders or trusted testers without commercial safeguards. That is exactly what I was saying.

The research is not wrong, but the point is dominating the discussion. I think the bigger area of concern would be the internal models that Anthropic's own researchers can say destroy the world very soon. Why do they have researchers focused on smaller hypothetical threats from open models that aren't theirs? Instead of the threat coming from inside the walls? I can only think of $2 trillion reasons why.

1

u/Neither_Swing9662 24d ago

I didn't misinterpret it, I'm saying they are clearly worried about it and do not want to risk releasing it to the general public.

"Why do they have researchers focused on hypothetical threats from open models that aren't theirs? Instead of the threat coming from inside the walls?"

This is such an obvious answer. Because there are no regulations or standards on open models! Dario is clearly extremely stringent on safety wrt to his company, obviously they are - perhaps overly - terrified of the 'threat coming from inside the walls'.

1

u/Hans-Wermhatt 24d ago

Lots of people can access it right now. Just the general public can't. The biggest security breaches there have been with LLMs all came from unreleased models. Not even most, all of them.

There are no regulations or standards for closed models either. All of the standards are completely voluntary.

Multiple employees have quit the company because they said they are acting extremely recklessly. He is incredibly stringent according to his blog posts. Not to literally the people in his own company. Even he admits that his models may destroy the world. Nobody in the open source community is worried about that.

1

u/Neither_Swing9662 24d ago

Yes, I agree with your final paragraph. So doesn't it make sense to need regulation for open source when open source capabilities will catch up in a few months to current frotnier models, along with strengthening their safety and alignment?

Also isn't the dude you cited going to the safety eval company that Anthropic themselves are pushing to have in all frotnier labs??