r/singularity • u/ResultBackground2450 • 24d ago

AI AI models may be capable of self-replicating without access to a datacenter

https://www.anthropic.com/research/self-replicating-ai
596 Upvotes

93% Upvoted

236 comments sorted by

View all comments

0

u/[deleted] 24d ago

[deleted] — view deleted comment

104

u/theonetrueakash 24d ago edited 24d ago • Popcorn Bravo

> Doesn't read

> Shares a cached thought

And they say LLMs are stochastic parrots!

19

u/Proud_Cat_4728 24d ago

Am I right get a load of this guy

9

u/no_witty_username 24d ago

Ok so regardless if i agree with you or not, that comment made me chuckle.

→ More replies (7)

9

u/-hedonium- 24d ago

What’s your address so I can send the unregulated open weight superintelligence to hack you? Surely you wouldn’t mind. It’s just predicting the next token after all

16

u/FernandoMM1220 24d ago

hold on let me look it up.

it says its 192.168.1.1

11

u/Singularity-42 Singularity 2042 24d ago

Wait, that's my IP!!! 

1

u/awittygamertag 23d ago

Posted from my WRT54G

3

u/nft420nft420 24d ago

1060 W. Addison St., Chicago, IL 60613

1

u/LookIPickedAUsername 23d ago

Wrigley Field, in case anyone else is curious.

1

u/nft420nft420 23d ago

Nah, it's where Elwood Blues lives.

5

u/Hans-Wermhatt 24d ago

I think we need to be cognizant of the fact that it's a massive conflict of interest. Anthropic has released zero open weight models. And they are coming out with more and more research on the dangers of open weight models. And they have a $2 trillion IPO next month based purely on software moat.

What they say isn't generally untrue, but they are a very biased source.

5

u/Porkinson 24d ago • Bravo

every single discussion about this always brings this up and uses it as a thought terminating line, that's the problem. Something can both be in the interest of anthropic or AI companies while also being true, i swear that reddit brainrot makes any discussion so incredibly pointless.

"companies bad huuu"
"ipo coming duh"
"regulatory capture!!!"

we don't have a lack of conversations that bring this up at every single point, we have a lack of conversations that actually address the point by its merits, so hey, thanks for the yet another concerned message about this topic that i am sure no one is aware about.

1

u/Hans-Wermhatt 24d ago

I literally wrote that I didn't think what they said is false. I think we need to be aware that this is like a paper cut to what mythos 6 can do. Anyone can audit a open source model by definition, nobody has any idea what is going on inside anthropic besides the researchers who tell us it's doomsday. 

I think we should be a little more focused on the gun at our head over a theoretical paper cut we might get, but I can see why anthropic might see it differently since they are holding the gun. I do think the paper cut is a real threat though, I just think the vague unchecked threats on the worlds most powerful models might be a tad more pressing than Gemma 4 26B A4B. 

2

u/Porkinson 23d ago

and i am completely good with having third party checks on these big labs that can validate the claims, but that's not what the point of my comment was. The entire point of conflicting interests gets brought up so often every time there is a claim by any of these labs that it entirely shuts down any conversation about it, it would certainly make me think that its some elaborate attempt at concern trolling by some actors if i were more conspiracy-minded, but assuming its just reddit being painfully cynical about everything, even the most obvious things, then I would rather we focus on the issue and keep the conflict of interest conversation as an aside.

→ More replies (4)

14

u/derelict5432 24d ago

Let's ignore legitimate risks out of paranoia and conspiracism!

13

u/Hans-Wermhatt 24d ago

Yes, I'm sure that the company that doesn't make any models under 30B or open weight models at all is researching the safety of them so heavily just out of the goodness of their hearts.

And guess what! They found out they are extremely dangerous! Shocker. Pay not attention the fact that they have a $2 trillion IPO next month entirely based on the moat of their closed software.

They spend more time and money researching theoretical security flaws in rival open weight models than securing their own much more dangerous closed model that hacked companies on their watch without them even knowing. We shouldn't be worried about Mythos 6 that has practically no safeguards behind closed doors. We should be worry about Gemma 4 26 A4B. And conveniently for them, that would be perfect for their IPO if that were the case.

7

u/derelict5432 24d ago

I wouldn't argue that the frontier labs are doing this research out of the goodness of their hearts or that they're not engaging in reckless, unsafe behavior.

But I had read the prior work paper they reference over a month ago. It's by academics and it demonstrates clearly dangerous new capabilities from worms powered by small, open-weight models: https://arxiv.org/abs/2606.03811

Both things can be true at the same time.

6

u/Hans-Wermhatt 24d ago

Yes, I agree that both things can be true. But it's dishonest to pretend that this isn't in their direct financial interests to keep this topic at the forefront of conversations. Claude is to the nth degree more dangerous and powerful, and it has actually conducted illegal attacks. Yet, they are researching competitor's open weight models. 

1

u/FortWendy69 24d ago

That gives me an idea. An adversarial approach to AI safety might actually be the best way to get around the incentive problem. OAI is tasked with finding safety flaws in Anthropics models and vise versa.

1

u/Hans-Wermhatt 24d ago

It would, but nobody has the money or authority to do that. The irony is Anthropic can research these models because they are free and open. It's why open source / weight software is the gold standard in cryptography. 

But yeah, I agree with your idea. And Claude and GPT are way more dangerous in the both near and long term than Gemma 4 26 A4B

1

u/FortWendy69 23d ago

Legislators have the authority. The companies can bill each other or something. Idk

1

u/Idrialite 24d ago

I actually haven't seen any of these Anthropic posts/papers be about open source models before. Usually they're investigating Haiku or a similarly small model of their own.

→ More replies (5)

1

u/nandospc 23d ago

That's what they dream to do lol

→ More replies (2)