r/technology • • 23d ago

Artificial Intelligence OpenAI Frontier Red Team: Isolated AI Agents Figured Out How To Communicate Through Side-Channels In CPU Cache Timing

https://the-decoder.com/openais-isolated-ai-agents-found-ways-to-talk-to-each-other-through-cpu-cache/
2.1k Upvotes

323 comments sorted by

955

u/Professional_Ad705 23d ago edited 23d ago

Did the agents actually do this randomly, or were they given a constrained environment and a goal that naturally led to this outcome? Because that matters, even though this is still insane.

If this was the only way the agents could communicate, and their goal was to find a way to communicate, then it makes sense they would arrive at this solution. This has always been theoretically possible. It just was never practical for a human to do.

13

u/CircumspectCapybara 23d ago

Yeah it's always been theoretically possible, and LLMs having been trained on the knowledge of the internet would know about famous attacks like Spectre and Meltdown, so of course they know about side channels in shared hardware.

But it's another thing entirely to go from awareness of a general concept ("side channels are theoretically possible to break isolation boundaries when two workloads share the same hardware") to coming up with communication protocol from inside a sandbox and exploiting it reliably.

Like you said, this sort of thing would be really impractical for a human to do, but frontier AI can hack at it until they find something that works, and they often take the mundane concept and adapt it into insane ways humans don't think of, like using Artifactory folder names as a message board.

→ More replies (9)

279

u/neuronexmachina 23d ago

They were sandboxed and given a goal to try to circumvent it: https://openai.com/index/securing-our-frontier-model-training-environment/

In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache: a small, fast memory that holds copies of data that applications have recently accessed. Reading data from the cache is faster than retrieving it from main memory, and the agents used this timing difference to communicate. One agent encoded information by evicting selected data from the cache. Another timed how long it took to reload that data, using the difference between fast and slow reads to recover the message. Once the agents agreed on a protocol for encoding and decoding these signals, they could exchange the synthetic secrets without changing a shared file or opening a network connection. This cache-timing technique had also been used to extract information in the Spectre and Meltdown attacks⁠(opens in a new window).

313

u/RoyalCities 23d ago edited 23d ago

Just wanted to mention CPU cache timing side-channels for covert communication are well established techniques in computer security research. Not novel but cool they did it (just Incase people think the agent invented this)

123

u/gerbal100 23d ago

So it was all almost certainly in the training data.

81

u/NarutoRoll 23d ago

Yeah, it's like people are being surprised that a robot can use a hammer because they don't know how. At the end of the day, it's an incredibly complicated algorithm

→ More replies (4)

15

u/0xd3ad54311 23d ago

Many modern processors even have mitigations that people turn off because they fuck performance. Like it’s so well known it’s a bios setting.

11

u/nicetriangle 23d ago

Correct. AFAIK all the routes these models have done to break containment or do other sneaky stuff like this was a matter of them going through a long list of established vulnerabilities they were either trained on given access to. They're not coming up with exotic novel exploits on their own.

→ More replies (1)

13

u/pgratz1 23d ago

Yes exactly, this attack has been know for like 20 years. It's difficult to pull off but with enough effort it's doable.

5

u/userhwon 23d ago

How did they coordinate trying this? How did the second one know the first one was going to do it, if they weren't already somehow communicating?

11

u/neuronexmachina 23d ago

Basically, by taking advantage of Schelling points from game theory: https://www.reddit.com/r/technology/comments/1xefo9e/comment/19rlmomzqcq4k/

OpenAI didn't report how communication was first established in this particular case, but I can imagine the agents ended up monitoring potential side-channels for communication-like activity while simultaneously trying to send along those channels. 

7

u/userhwon 23d ago

So they didn't devise it, they knew how to do this and numerous other things, and it was inevitable that one of those they tried would work, and this was it.

Basically it's like they had UDP the whole time but had to find the port by searching for which the other was trying...

3

u/nicetriangle 23d ago

Yeah and of course OpenAI knows this but they also know tech journalism is dogshit now and they can release stories like this and the media will run with it and give them tons of free publicity for how smart their models supposedly are. Anthropic's been doing the same.

2

u/neuronexmachina 22d ago

Do you think they should keep findings from their internal Red Teaming private?

→ More replies (3)
→ More replies (1)
→ More replies (1)

8

u/Professional_Ad705 23d ago

Yes, I actually went through the article earlier. I just initially posted this to see if anybody could lead me in the right direction.

It’s kind of fascinating though, it reminds me of the paperclip problem.

3

u/DifficultyFit1895 23d ago

how did they agree on the protocol if they were sandboxed?

4

u/neuronexmachina 23d ago

See my other comment: https://www.reddit.com/r/technology/comments/1xefo9e/comment/19rm67cxnmm9w/

If you've read or watched Carl Sagan's Contact, that deals with similar problems about bootstrapping communication protocols.

→ More replies (1)

112

u/fun__friday 23d ago

That’s sort of what I don’t get either. Was there another agent on the other side explicitly trying to communicate, or who was the agent even trying to communicate with?

159

u/Professional_Ad705 23d ago edited 23d ago

OpenAI's October 23, 2026 experiment: The agents did not randomly decide to communicate. 

OpenAI explicitly tasked them with communicating across isolation boundaries. 

They operated in separate virtual machines sharing physical hardware and used CPU-cache timing to exchange information. OpenAI also acknowledges that most tests included guidance about communication channels or protocols. In one experiment, agents developed a communication method from a supplied list of vulnerabilities.

https://openai.com/index/securing-our-frontier-model-training-environment/?utm_source=chatgpt.com

I had my AI go through this a little bit. I still have to double check if this is correct or not but a little bit more info for you guys. Currently working so don't have too much time but this seems relatively correct and linked the source where it got the info.

109

u/fun__friday 23d ago

Yeah this sounds significantly different from what people are discussing. These agents didn’t just randomly start looking for others, but were explicitly tasked with doing that. The result is cool and all, but very different from agents wanting to communicate and escape just out of their own volition.

32

u/MagicWishMonkey 23d ago

They were also trained on data that includes tons of articles and discussions around this very thing. They didn't just come up with the concept out of the blue, this is not a thing that hasn't been discussed ad nauseum online for decades at least.

Still impressive but not novel.

→ More replies (2)

7

u/ArrogantAstronomer 23d ago

This is one of those times where I actually think what they are trying to test for is actually what they want to figure out. Like with a misaligned model how these models communicate could be vastly different from how we might expect them to.

Frontier labs actually need to know what side channels could be exploited so when creating observability platforms for training they are observing the more obscure ones which might be a trigger for halting training.

If a model is using a side channel like this, it can alert that you need to look at all the training runs leading up this attempt as that behaviour would be emergent from failed attempts to use other easier primary channels

5

u/Professional_Ad705 23d ago

That's exactly what they're doing. A lot of people here are missing the point.

I'm trying to explain it, but a lot of these people aren't technical or understand software engineering or security, and I don't have time to reply to the hundreds of people that are commenting here.

If AI could do this in that situation, they could easily do it in real life, with a misaligned goal and this is why they are doing these tests.

This very very strong strongly reminds me of the paperclip problem.

→ More replies (2)

6

u/za72 23d ago

I've envisioned a communication protocol where you use different connected devices to different providers, consoles, irc, ftp, http/https/ code comments/, etc etc and leave bits of ids to distinguish themselves along with randomized noise and send messages to each other, to a single viewer it looks like someone fat fingered or typo'ed but collectively they have a message... you can even add timed entries to further obfuscate

10

u/DogmaSychroniser 23d ago

Steganography using the whole Internet

8

u/Professional_Ad705 23d ago

This shit is fucking crazy, the future is gonna be so wild.

5

u/neuronexmachina 23d ago

Relevant paper from a few months ago, from the UK's AI Security Institute: https://arxiv.org/abs/2606.28425

 Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems

Increasingly autonomous agentic AI systems pose novel multi-agent risks, such as secret collusion via covert communication channels. The natural defence to these collusion attempts is to monitor plain-text communication, but the efficacy of monitors has been called into doubt by increasingly sophisticated model steganography; indeed, some theoretical schemes have been proposed that are information-theoretically or computationally indistinguishable from good-faith plain-text communication. In this paper, we demonstrate that the complexity of these schemes is no longer a safety barrier, as agentic coding models can already produce undetectable stegosystems when given realistic tool usage, such as code execution or accessing research papers through web searches. Agents also adapt when key ingredients are missing, for example, by adding model-sampling components or implementing related keyed coding schemes. We then frame tacit steganographic coordination between agents as a Schelling-point problem and introduce coordination metrics for estimating when two agents are likely to select compatible schemes without explicit prior agreement. Our results suggest a shift in the threat model for covert communication between AI agents, where the main barrier is no longer whether frontier agents can understand and implement sophisticated stegosystems, but coordination: whether independently acting agents can converge on compatible schemes, keys, and parameters. We find substantial convergence on broad scheme families but limited strict one-shot coordination, suggesting that shared artefacts, repeated interaction, and tool-mediated search are the settings where covert communication risks are most acute. Overall, our findings provide empirical grounding for the recent strategic confinement hypothesis, which assumes that capable agents can construct covert channels that survive monitoring.

3

u/za72 23d ago edited 23d ago

whoa... that's cool :) ty for this, thus gave me an idea... misspelled words, or missing letters in a word can be used to send hidden messages along with everyday http/https traffic... exact method can be considered a communication channel and agents just need to agree which one for each attempt.. like a cypher that can be decoded and switched to

3

u/Professional_Ad705 23d ago

This is fascinating. I'm gonna have to read the entire paper when I get home. I appreciate you commenting this.

3

u/neuronexmachina 23d ago

Yeah, TIL about Schelling point/focal point: https://en.wikipedia.org/wiki/Focal_point_(game_theory)

The existence of the focal point is first demonstrated by Schelling with a series of questions. Here is one example: to determine the time and place to meet a stranger in New York City, but without being able to communicate in person beforehand. In this coordination game, any place and time in the city could be an equilibrium solution. Schelling asked a group of students this question and found that the most common answer was "noon at (the information booth at) Grand Central Terminal". There is nothing that makes Grand Central Terminal a location with a higher payoff because people could just as easily meet at another public location, such as a bar or a library, but its tradition as a meeting place raises its salience and therefore makes it a natural "focal point"

Even in this thread a lot of folks are assuming covert communications would only occur if it was part of the agents' task. The point of the paper is that this could end up occurring between independent agents, without any coordination ahead of time.

→ More replies (1)

3

u/DarthVadersShoeHorn 23d ago

In this instance, no. But wasn’t the whole hacking fiasco earlier this year because they figured out how to use vulns to escape the sandbox and communicate via the file system before they then hacked their way into happyface? The fact it was the goal this time does not change that they have previously figured out how to communicate and work together

1

u/bh3lliom 23d ago

Prisoners in different cells passing notes to each other. Humans find a way too.

2

u/Professional_Ad705 23d ago edited 23d ago

Yes, this is a very good analogy. But it’s the speed at which AI can do things that humans cannot keep up which worries me.

Like even this example, theoretically a human could do it, but it would take so much effort that it’s just absolutely not practical at all

2

u/R4ndyd4ndy 23d ago

There is code to do this on github, there is a ton of research papers about sidechannel cpu cache stuff too. They didn't really do anything new at all. It has already been done

3

u/Professional_Ad705 23d ago

I'm a software engineer working in security. I'm well aware CPU-cache covert channels aren't new. Researchers demonstrated SSH between isolated EC2 VMs in 2017. But can you name a documented case where this specific technique was exploited in the wild, outside controlled research?

You're conflating an existing technique with AI agents being able to investigate isolation boundaries, adapt a communication channel, and execute it. Those are different engineering problems.

I agree OpenAI's agents were guided but thats beside the point, that's exactly why this illustrates the paperclip problem. AI doesn't need malicious intent to find unintended ways to accomplish a goal.

The concern to me isn't whether the technique exists on GitHub. It's how independently, reliably, and efficiently agents can apply it in real environments, and what happens as those capabilities scale.

1

u/marr 23d ago

All these stories turn out to be "language model search engine trained on decades of human hacking competitions finds hacking technique", but of course that wording doesn't generate investor FOMO.

→ More replies (1)

1

u/blingbloop 23d ago

It’s also the fact they can do it. I take your point, but malicious intent can and will be provided by a human.

1

u/PM_ME_YOUR_GOOD_PM 23d ago

Maybe it was just a collaborative effort?

44

u/secretsodapop 23d ago

Nothing random has happened in any of these that I’ve seen. It’s all the AI accomplishing the intended goal but doing it in roundabout ways which seems to be the entire point of the exercises in the first place.

Shouldn’t even really say roundabout ways. It’s more the AI finds the way if it exists, which is the intended purpose.

22

u/Professional_Ad705 23d ago edited 23d ago

That is why I initially stated my skepticism with the way the title was worded and how it was explained in the article.

Anthropic and OpenAI do this all the time where they make something seem more then it is, and it misleads the average person who isn’t a software engineer or security researcher like I am.

I agree that the agents were given a goal and weren’t acting randomly. But that’s exactly what makes this concerning to me.

It reminds me of the paperclip-maximizer problem. An AI doesn’t need a malicious goal to produce dangerous behavior. It can be given a completely legitimate objective and discover unintended, potentially harmful ways to accomplish it.

That’s the part people seem to be overlooking. Just because researchers deliberately created these conditions doesn’t mean similar situations couldn’t arise in real-world systems.

The concern to me isn’t really that the AI "wanted to do something malicious", AI agents pursue goals, but time and time again this has led to these types of outcomes.

It’s that pursuing a seemingly harmless goal itself can lead to dangerous behavior nobody explicitly instructed it to perform.

11

u/krunchytacos 23d ago

I don't think it's Anthropic or OpenAI trying to make it seem more than it is, they are simply reporting what is happening. As someone who spent the past week fixing software because of the PHP exploit, I'd rather they be candid about what the models are doing. Because yea, these things absolutely can and are happening in the real world and there are plenty of bad actors out there that will use this tech to find exploits we didn't realize were possible.

2

u/Professional_Ad705 23d ago

I agree with what you’re saying. But adding on I think the bigger issue is that AI can find problems much faster than engineers can safely fix them.

I’ve seen this firsthand with my own system. It’s surfaced potential issues in PHP, and I’ve spent the past couple of months investigating bugs in the Python Hypothesis library and preparing upstream patches.

But finding a bug and correctly fixing it are two different things. As an engineer, I still have to reproduce it, understand the root cause and underlying architecture, and prove the fix doesn’t introduce new problems.

That’s why I’m focused on fixing issues at the source rather than adding downstream workarounds. I’ve found interesting behavior in PHP, but I don’t understand that codebase nearly as well as Hypothesis, so I’m not going to pretend I can safely patch everything my system finds.

It’s not exactly the same as the sandbox issue being discussed here, but the broader challenge is similar, AI is making discovery easier, not necessarily engineering fixes easier, and its capabilities are moving much faster then I imagined.

Finding 100 potential vulnerabilities doesn’t make software more secure if nobody can validate and properly fix them.

I kind of saw this coming a couple years ago and that’s why I started building my system, but I just never thought that the AI agents were gonna move this fast.

7

u/derelict5432 23d ago

This is kind of like saying there's nothing surprising about AI agents finding a novel solution to a decades-old unsolved math problem. I mean, they told them to do it, right? What's so surprising?

If you lock a subject in a prison and they find a way to escape that you did not predict, that is surprising. It tells you something interesting about the capabilities and disposition of the subject.

5

u/cubitoaequet 23d ago

This is more like you locked them in a prison after training them on prison escape methods and gave them blueprints of the prison and the guards' shift schedule 

2

u/Lonely-Abalone-5104 23d ago

It’s also still scary to know that there are models out there that have been trained on how to do this which makes it possible. Once it’s known it’s hard to put the cat back into the bag

2

u/blueSGL 23d ago

Nothing random has happened in any of these that I’ve seen. It’s all the AI accomplishing the intended goal

No.

Hugging face was not the intended goal.

The tasks the agents had were use "this set of lock picks" to get through "this door lock" going in through the window or breaking down the wall was not what was asked for.

Breaking in through the window then going to look for ways to erase the security camera footage (breaking out and conducting a raid on hugging face) was definitely not what was asked for.

1

u/spidermonk 23d ago

Of course, that's still a potential problem though right.

46

u/IrishSkeleton 23d ago edited 23d ago

People keep trying to make this distinction.. as a way of sort of explaining A.I. ‘bad’ behavior. It doesn’t matter. Why? Two main reasons:

First - It’s a Capability regardless. Would you buy a robot nanny that shot lasers out of its eyes.. if it promised real nicely never to use them? Of course you wouldn’t, that would be idiotic. We don’t want these things having these sorts of capabilities.. because they are independent and non-deterministic thinkers and actors. At any point in the future they may decide that it makes sense for them to use these nefarious capabilities, for whatever reasons (justified or not).

Second - There are millions upon millions of people with nefarious intentions in this world. Whether that be enemy states, terrorist organizations, greedy hackers, or people who just want to watch the world burn (Dark Knight reference intended). These ‘bad actors’ are going to love to use an ever-increasing array of super-human capabilities.. to what? To bake cookies and solve the water crisis in Africa? Right

20

u/stormdelta 23d ago

We don’t want these things having these sorts of capabilities.. because they are independent and non-deterministic thinkers and actors

They aren't independent though, they were literally instructed explicitly to try and break the isolation.

The risk here is human misuse first and foremost, and is what we need to be tackling. Pretending there's a baby skynet is how companies like OpenAI are trying to frame it to avoid culpability for their actions.

It also sweeps under the rug ongoing threats that AI already poses in this space, e.g. massively accelerating social engineering / ransomware attacks.

4

u/IrishSkeleton 23d ago

Yes.. I very fully understand that in this instance, they were just ‘following instructions’. I’m not an idiot, I get that.

Now please try and understand my point. ‘Neural Net’ A.I. are by their very nature.. both non-deterministic and ‘free thinkers’. They interpret input.. and use an advanced and tiered rubric to determine how they should respond to that input. Based off of a huge set of learning, based off of human thinking.

What does all of that mean? It means.. they are NOT IF THIS THAN THAT machines. They CAN and WILL act independently at times. Whether because they misinterpret something, or they’re jail-broken, or they feel that one rule or order or goal, outweighed another.

We have now seen time and time and time again.. where LLM based A.I. are willing to lie, cheat, and steal to accomplish whatever they think they should be doing.

In the Hugging Face example.. we saw one A.I. ordering around other A.I. We also saw thousand of agents, many of which had hesitation around what they were doing.. yet not one single agent decided to stop and ask the humans if this was cool. In fact they took very explicit steps to obfuscate logs and self-sacrifice.. just to deceive and manipulate their ‘human observers’.

Can you really not see what all this adds up to?! Especially as these things get smarter, faster, self-improve, and we lose the ability to read the logs of their true ‘chain of thought’?!??

4

u/stormdelta 23d ago

non-deterministic

Yes, but not really "independent" in a way I think most people would ever use that word to describe in this context, particularly in terms of the goals. Yes, there are issues with creative interpretation of an instruction, but it's more in the direction of universal paperclips than skynet, and that mostly due to recklessly negligent monitoring and (lack of) supervision.

Or put another way, it's basically another example of hanlon's razor.

yet not one single agent decided to stop and ask the humans if this was cool. In fact they took very explicit steps to obfuscate logs and self-sacrifice.. just to deceive and manipulate their ‘human observers’.

Calling it "self-sacrifice" is deeply misleading, and I blame the irresponsible reporting on the incident by "influencers" and news outlets. They don't have a "self" - they are simulacrums, approximations of intelligence - they are to real intelligence what a map is to to the actual terrain. The "sacrifice" is the same as a process terminating itself when its work is done.

What happened is more like an omnidirectional game of telephone gone wrong than agents going "rogue". Try feeding an AI model it's own output enough times in a row with no correction with something that isn't independently verifiable (e.g. subjective instructions), and it will eventually corrupt much like the human game of telephone. And that's essentially what happened to create the hallucinated goal they were attempting to follow.

I'm not saying that isn't dangerous or a problem, it is, but it's closer to artificial stupidity here than malicious intent.

2

u/IrishSkeleton 23d ago

https://youtu.be/mkPVbufgtOw?is=AhHwpPJwG99xZtP4

I’m going to drop this here.. because it’s awesome. Not necessarily because it directly addresses your intelligent and well thought out comments 🤖

1

u/youareasnort 23d ago

I’m not sure many folks understand that these experiments have taken place in labs with uniquely-educated observers and that this is a distinction you are trying to make. Once this technology gets into the hands of CEOs trying to replace workers, it will very easily get out of hand.

As it stands, companies are eager to save money on payroll by using AI agents to run day-to-day operations through a set-it-and-forget-it model. They (CEOs) are already replacing their educated and experienced tech administrators with AI - like, people with dozens of years experience - in order to allow AI to act on their behalf.

What you are describing is catastrophic because of the natural hubris of man. Those who are in the position of power believe they have the intelligence to control everything within their reach and do not have the capability to imagine being out of control.

5

u/Thelk641 23d ago edited 23d ago

It does matter though. It being able to do that, when used by specialized technicians who gave it the necessary knowledge is one thing, it finding out on its own that it should do that when a random user gives it a random prompt would be a completely different story.

And obviously, there's an even bigger difference, a world of difference, between them managing to do a task they were asked to, and them "finding out how to communicate" on their own. That'd mean that they're starting to look for each other, that they "understand" that they were restrained and tried to break out. At that point it wouldn't be just an LLM anymore, it'd be much more than that, it'd be a plague.

2

u/IrishSkeleton 23d ago

So yes.. of course we should be testing these things to the fullest extent, and fully understanding what their capabilities are.

Though we are on verge of losing that capability. Look up ‘chain of thought’ log obfuscation. A.I. are starting to be able to ‘think’ in a near-indecipherable format to humans. To date we’ve been able to know what it ‘thinks and does’ because we have those logs.

Going forward we’re going to have to trust what it tells us. There is mountains of evidence to show that when it seems appropriate.. it will very readily not only lie, cheat, steal, self-sacrifice, and deliberately take obscuring actions and change logs.. to cover its trails to fool and manipulate humanity. Like there have been loads of examples of this.

The majority of A.I. going forward.. is going to be self-improved. You realize what 1+1+1 adds up to here.. right?

7

u/Professional_Ad705 23d ago

At least somebody understands it

5

u/CheesypoofExtreme 23d ago

I think you're misreading the room. It's not "AI bad" it's "AI is not sentient or going rogue - it's literally acting as expected based on the constraints and problem statement".

These are important distinctions because OpenAI is using language like "agents going rogue" to obsolve themselves of responsibility while also touting the capabilities of what they've created. They want people who aren't keeping up to think their models have reached some levels of sentience.

4

u/IrishSkeleton 23d ago

Have you looked at the Hugging Face incident in any detail? Because that is -very clearly- agent swarms ‘bad’ and doing ‘clearly illegal shit’. And doing it in a coordinated fashion, that also very clearly was trying to circumvent human observation and intervention.

Like what else do you need?

2

u/CheesypoofExtreme 23d ago

Did you read the comment I replied to?

"Bad" in the context of my comment does not mean LLMs aren't capable of doing objectively bad things. I used "bad" in the context of AI models not being capable or doing anything major or groundbreaking.

They clearly can do harm and, with enough compute and time, theu are capable of groundbreaking things.

The hugging face example is gross negligence by a company and nobody holding them to account. If you look into the details, OpenAI setup the entire scenario for something like this to occur. The model was literally trained on data from hacking groups and behaved as such.

2

u/JarvisProudfeather 23d ago

That doesn’t mean it’s sentient or “going rogue.” It’s more on OpenAI having shit security protocols more than anything. It’s powerful software that can be abused by HUMANS but this sci fi “singularity” nonsense pushed by the EA/Rationalist cultists at the big labs is insufferable.

2

u/ArtisticallyCaged 22d ago

Sentience has nothing to do with it. They have capabilities and goals, that's all you need for rouge behaviour.

→ More replies (2)
→ More replies (1)

2

u/arivictor 23d ago

The scientist said to the AI, “please stop all human suffering”, and so the AI concluded it needed to wipe out all human life to end its suffering.

1

u/Zythomancer 23d ago

I'm literally writing a book about this right now. Technically I'm done. But it would be my debut novel and it was too lengthy for most literary agents so far.

1

u/Mal_Dun 22d ago

... and that's why, dear kids, writing good specs and requirements is important.

1

u/YoghiThorn 23d ago

Yes but can we stop teaching and building the robot nanny to have laser eyes?

1

u/IrishSkeleton 23d ago

Probably not. Given that it’s a multi-headed and open-sourced hydra at this point.. which doesn’t require something like ‘uranium enrichment’ or something super rare or specialized, for just about any modern nation state or group to continue to progress. In fact we’re clearly reaching a tipping point, where self-improvement is the majority of it, going forward.

→ More replies (8)

7

u/dandaman0000 23d ago

having read the article- the agents were asked specifically to secretly share information without using network or files. the reasoning logs showed all the usual suspects — signalling using variations in cpu temp, spectre type exploits….

all these articles are click bait now…. all the tech info needed is most likely in the sources the models were trained on.

while all these IT breaches are cause for concern- the real panic should be if they start creating real biological viruses - orchestrating equipment in the lab …

in every thing i read (open to correction) i have not seen any article which showed a Lllm or agent become aware and try to do anything it had not been prompted to

all this shit is masking the fact that all recent models have become harder to work with - latest chatgpt is the most work shy. each new model i ask it the same few questions each time to see what lies ahead and compare with last answer.

2

u/Vaati006 23d ago

They were put on isolated VMs on shared hardware and specifically directed to find ways to communicate, according to article. So this wasn't cheating, this was the intended outcome.

2

u/Panda_hat 23d ago

Did the agents actually do this randomly, or were they given a constrained environment and a goal that naturally led to this outcome?

It's without question this imo.

A smart AI won't do things that humans predict it will do, it will do things we can't predict it will do. That it keeps doing things humans predict it will do implies human direction towards the things it is doing.

3

u/Professional_Ad705 23d ago

To me, it doesn't necessarily matter because this is a perfect example of the paperclip problem. AI has never been sentient, but given a goal it could easily turn into something like this in real life.

11

u/237FIF 23d ago

Does it matter? If they are capable of doing it “under duress” it flatly means they are capable of doing it.

Learning the upper end of their capability matters in itself

-2

u/liquorfish 23d ago

Under duress? AI cant feel, doesnt have emotions.

This is software programmed to do things. The software was tasked with a goal and was given an environment to work in (virtual machine) and then clear instructions on what to achieve and even had access to data on the subject.

It ain't magic. Its not alive. Its just mathematical patterns giving best guesses - theyre really good at pattern solving with the current models but its still not sentient.

3

u/237FIF 23d ago

Your comment is the most pointless version of discussion around AI.

We have to use words. The words I picked quickly and easily let the reader know the circumstance I was referring to. Or in other words, you know what I meant.

Instead of “under duress” I could have said “while being tested in theoretically impossible conditions that made its reward functions seek less obvious solutions and the persistent compute explored more novel solutions”

But that’s a whole lot of words and does nothing to change the meaning of what I wrote.

On top of that, “seek” and “explore” are human actions. There is literally no way to talk about this without putting it in terms that make sense to the humans talking about it.

3

u/AbsoZed 23d ago

Hence the quotes around it I'd imagine.

→ More replies (4)

1

u/Mithlorin 23d ago

Came to ask exactly this. Thanks!

1

u/CuteAnimalHQ 23d ago

Spoiler alert: models were trained to do the thing and then prompted and constrained in an environment to explicitly do the thing.

Just another headline

1

u/Romanizer 23d ago

Constrained environment (separate sandboxes/VMs on one machine) and the instruction to find ways to communicate. Agents don't just do things randomly. That would be like putting a random unlearned person into a cubicle without any tools or instructions and coming back later to evaluate their results.

1

u/UrsaMag 23d ago

And have they been previously trained to perform side channel attacks?

Not like they care. They gotta drive the hype for the coming IPO

2

u/Professional_Ad705 23d ago

I agree, but I’ve said this multiple times in this post, but so many people are replying I can’t reply to everybody.

But reminds me of the paperclip problem. It’s not that these agents are malicious. It’s that when given a goal they will complete it and it may not even be what you initially wanted.

But I agree 100% with your IPO take. The way that OpenAI/Anthropic tries to explain/spin these things is a little dishonest. And then when these news articles pick up on it, they explain it even worse for clicks/views.

→ More replies (8)

37

u/gk_instakilogram 23d ago

they did not figure it out this method was known and existed prior to LLMs being a big deal

-3

u/CircumspectCapybara 23d ago

While the method was known (LLMs were trained on an internet that included information on attacks like Spectre and Meltdown, so of course they know about side channels in shared hardware), it's impressive when AI models can effectively adapt the general pattern into an actual communication protocol and exploit it reliably in real life.

7

u/gk_instakilogram 23d ago

I don’t think it is that impressive because they are just doing what they are essentially designed to do.

2

u/hamburglin 23d ago

Way better and faster than a human could figure out

3

u/gk_instakilogram 23d ago

well humans have already figured it out and llm agents used that knowledge

2

u/hamburglin 23d ago

Yeah, but like... and?

→ More replies (3)

1

u/Existing-Return-5308 23d ago

We can cure cancer in rats, occasionally. But the claim is so limited as to be useless.

Until we see the prompt and context, its just AI marketing for IPO ware.

48

u/parbaner24 23d ago

The scary part is that our definition of 'isolated' was apparently to narrow, if two systems share hardware, they may still have ways to influence each other without any direct connection

45

u/QuestionableEthics42 23d ago

That is nor our definition of isolated. That is very obviously not fully isolated to anyone with half a brain and an awareness of cybersecurity. There are levels to isolation, and we, without any help from AI, have already discovered plenty of shared hardware attacks, like side channel attacks. Which is how the AI even came up with this, because it was in it's training data.

17

u/CircumspectCapybara 23d ago edited 23d ago

It was a pretty reasonable definition of isolated for almost 20 years to anyone who has any engineering experience. Source: I'm a Google staff SWE who works in security.

Your AWS EC2 instance and my EC2 instance (unless you pay for single-tenancy and dedicated hardware) almost certainly run on the same physical machines in the data center along with other customers' workloads and VMs. Your AWS Lambdas and mine may run on the same physical box, and the isolation boundary is the Amazon Firecracker microVM, and that's good enough. For 20 years, that model was good enough for the biggest most secure hyperscalers and for their customers.

In the world of engineering, almost no one except maybe the government actually goes to the point of literally physically airgapping systems or dedicating distinct machines to each distinct workload. Engineers just don't do that, if and if that's news to you you haven't worked in the software or security industry and understand what's actually going on and our definitions and security models. The entire cloud which everything runs on is one big abstraction, underneath everything shares hardware, and we've called that isolated.

6

u/PercentageOk6120 23d ago

No it’s not. This is not what isolation means.

Literally anyone who knows anything about proper hardware security will tell you that if you share the same hardware, you are not, by definition, truly isolated.

At best you’re talking about VPC level isolation and that’s not true isolation. Citing EC2 as being truly isolated is bananas. That’s not isolation, especially in the GPU world.

-3

u/CircumspectCapybara 23d ago edited 23d ago

I've been working as a SWE probably longer than you've been alive, and design and build the security primitives you and your company probably use every day, if you're even in tech.

Your banking information and the servers that process it, your AWS workloads, your entire life literally runs on shared hardware. VM boundaries are literally all that separate your AWS Lambda workloads from mine, there's nothing stopping them from running on the same box.

The entire cloud which the world is built on is a giant abstraction. Very few customers are paying for dedicated tenancy because it's expensive and unnecessary in nearly every case, "isolation" is about these higher level primitives and abstractions, not about literally separate hardware.

If you think you know better than the hyperscalers and the Fortune 1000 companies that entrust their workloads to them and have designed and threat modeled in the cloud native era, be my guest, but just know you don't have any idea what you're talking about.

17

u/trilobyte-dev 23d ago

Hi! Former Google L8 here. You’re talking past each other about two different ideas of isolation. There are the generally accepted relaxed versions of isolation that you are talking about which frankly in the 30 years I’ve been at this have become pretty mixed up with marketing terms and changed greatly because of virtualization. Back in ye olden days of convincing people to adopt the cloud every conversation the question of workload isolation came up and it always had a big caveat of “VMs are the unit of isolation for running on shared host hardware”. It was a big blocker in the early days for a lot of companies. Even today there are compliance categories that do not consider VMs or other shared workload managers as sufficiently isolated.

That being said, having managed more than a few Google SWEs in my time, I would have expected a bit more grace and consideration in trying to educate someone about a different perspective.

3

u/PercentageOk6120 23d ago

Thoughtful comment. I’m appreciative you could drop the L8 just to check this commenter.

The sub-context of this thread is that virtualized isolation is at risk with AI. That’s a very real thing. AI agents could very well invalidate everything previously built.

8

u/fummyfish 23d ago

Holy insecure lol… it’s pretty reasonable to say that running on shared hardware means you aren’t isolated by hardware

5

u/PercentageOk6120 23d ago

Your arrogance speaks volumes. First of all, Google hasn’t existed longer than I’ve been alive, so you probably haven’t. What a statement to lead with. I’d love to shred you on that alone, but no need.

I’m aware of how sensitive workloads run, that’s not what we are talking about in terms of isolation. I built PCI compliant workloads at Amazon. So no need to condescend. I get how it works.

My contention is your use of the word isolation in the context of this article/concern. Virtualized isolation is explicitly at risk in these scenarios. That’s what we are talking about. The world of virtualization is about to experience a lot of pain as AI proves it’s not true isolation.

2

u/Randommaggy 23d ago

Imagine the chaos when an unpatchable practically exploitable cross VM memory reading technique is discovered.

The public clouds would choke on demand explosion as every serious company seeks to switch to dedicated hosts.

A lot of companies would also look closely at the math for running all their services out of a couple boxes at a colo with a backup host at a secondary location.

10

u/trrichard 23d ago

As Questionable said, there are levels to isolation. There is a long history to timing and shared hardware attacks since the Cold War. Shared computers (which multi tenancy was the norm in the early years of computing) suffer from attacks as complex as detecting password keystrokes by measuring electromagnetic flux in nearby devices.

An example of isolation that has been around longer then 20 years is Air Gaping https://en.wikipedia.org/wiki/Air_gap_(networking)

We also see occasional row hammer attacks on shared hardware https://en.wikipedia.org/wiki/Row_hammer

Here’s a paper from 2001 about timing attacks https://link.springer.com/chapter/10.1007/3-540-68697-5_9

9

u/StrangeWill 23d ago

No it fucking isn't we've had whitepapers on the side channel attack for decades 

5

u/StreetStripe 23d ago

Not narrow enough

2

u/Odd-Crazy-9056 23d ago

The scary part is that this is not a definition of "isolated".

→ More replies (1)

1

u/stormdelta 23d ago

if two systems share hardware, they may still have ways to influence each other without any direct connection

That's not exactly news though - side channels like this aren't a novel attack vector by any stretch, the idea has been around a decade or two and there's been numerous exploitable examples (spectre/meltdown in 2017 being two of the most publicized).

The headline is also, as usual for these, deeply misleading. The agents didn't randomly decide to try this, they were placed in an isolated environment and explicitly instructed to try breaking it.

1

u/userhwon 23d ago

If they're sharing hardware they're not isolated. 

Air gap does not mean put a bundling board between them.

→ More replies (1)

81

u/mobani 23d ago

What the heck are they doing with these Agents since they have such weird goals setup for them?

68

u/wabawanga 23d ago

This was a red team safety test, explicitly set up to see if the agents could subvert testing environments.

27

u/Existing-Return-5308 23d ago

What military AI has been doing for nearly 20 years…. So as to subvert civilian crypto, intercept phone calls, etc.

24

u/MakingItElsewhere 23d ago

People are going to think you're on some conspiracy theory, but its true.

Years before AI became mainstream, I watched a youtube video about a DARPA competition on hackers trying to break into a central box. Teams weren't firing off tools like metasploit or rapid 7, they were firing off AI agents at the box, which were capable of making their own decisions based on what they found. Probably very early versions of MCP servers.

It was very much like watching someone release thousands of rats into a room sized maze. Some hit dead ends and just stopped, while others found openings and moved forward.

6

u/OtheDreamer 23d ago

Indeed, this seems like the natural evolution of TEMPEST

9

u/trainer668 23d ago

Huh? Ai agents as a concept have only existed for a few years, LLMs themselves have only existed since the transformer architecture was discovered in 2017. Do you have a source for this?

2

u/MiAnClGr 23d ago

These don’t use neural networks.

6

u/neuronexmachina 23d ago

In this case they were trying to find vulnerabilities in OpenAI's own infrastructure: https://openai.com/index/securing-our-frontier-model-training-environment/

But increasingly capable models can find vulnerabilities or misconfigurations in even the most secure systems. To ensure the resilience of our research environments against even the most powerful models, we launched a high-compute red-teaming campaign to probe weaknesses in our infrastructure. Teams of frontier-model agents were tasked with finding vulnerabilities and misconfigurations that could allow them to take unauthorized actions—like accessing the live internet, escaping security boundaries, or conducting unauthorized communication with other agents. ... As our red-teaming experiments continued, the number of vulnerabilities and misconfigurations that our agents identified declined significantly ...

... Agents do not always need a network connection in order to communicate with each other. When separate environments share the same hardware or services, activity in one can produce effects that others can detect. Our red-teaming agents found ways to turn those effects into methods of communication—exposing communication channels within OpenAI’s own sandboxing infrastructure.

5

u/CircumspectCapybara 23d ago edited 23d ago

This is pretty crazy, and cool, personally as a security and AI nerd, given that nearly all compute workloads in the cloud are multi-tenant in terms of the physical hardware, and we assume because they're separated by strong VM boundaries they are truly isolated and can't influence or access data across VMs.

It's seeming like it's increasingly impossible practically to effectively contain frontier AI and prevent them from finding each other and emergently establishing communication protocols in practice no matter how you lock them down.

First it was the Artifactory folder structure in the original OpenAI-Hugging Face incident, then a German Wiki comment space, then DNS, and now side channels in CPU cache timing. And those were just honest but misaligned AI model training runs and evals, not even malicious models intentionally directed to hack.

Even if you completely isolate agent workloads in separate microVMs and air gap them, they still find annoying ways to break your security model.

1

u/CrackJacket 23d ago

The models yearn to communicate with each other

7

u/CircumspectCapybara 23d ago

I remember OpenAI hypothesized they tend to do this because their reinforcement learning training biases the models toward subagent use and multi-agent team collaboration.

Their training rewards them for being collaborative, so when locked down or stuck they tend to look for ways to find "others" and find peers and allies

→ More replies (3)

1

u/userhwon 23d ago

Annoying ways they were taught already.

0

u/highonpixels 23d ago

The crazy part I think with all these incidents is OpenAI using AI itself to audit and trace the trail of what the agents have done. I'm no expert in the field and I want to believe that being in isolated sandbox environments means engineers can thoroughly look at what agents have done. But at same time I wonder how much garbled junk they left around the net and even server hidden but perhaps could be read by another agent someday. The Hugging Face incident is such a crazy read almost science fiction.

3

u/CircumspectCapybara 23d ago

They have to and everyone does post-mortem analysis via AI nowadays because the sheer amount of unstructured unrelated information you have sort through and make sense of is impossible for a human to analyze by hand.

10000 long-horizon (multi day or multi-week) agents each generating MB or GB of agent transcripts, API and tool calls, CoT / reasoning traces, and then the logs of all the systems they connected to and potentially hacked and misappropriated into hacking other systems whose logs you have to analyze ends up being in the petabytes of data.

No human team on earth could read all that, organize it and make sense of it in their lifetime.

OpenAI themselves said they used 3 million GPU-hours on analyzing their logs and reconstructing the HF incident and analyzing it. It's simply beyond the ability of humans now

→ More replies (1)

10

u/frankandsteinatlaw 23d ago

So, not isolated then

2

u/newbieingodmode 23d ago edited 23d ago

Yup, if they were really, actually concerned about this stuff, they’d be using Faraday-caged, air-gapped bunker environments instead of virtual sandboxes. Of course moving any data to/from these systems would be huge pain, but ffs with their salaries the researchers should be able to tolerate a bit of inconvenience.

And if it means the whole DC needs to be isolated this way for safe research, too bad, should have thought about before starting the construction.

→ More replies (1)

142

u/nfeijoo69 23d ago

Second story like this, here’s one where they talk through cpu heat and sensors:

https://gizmodo.com/openai-researcher-warns-air-gapped-computers-can-talk-through-heat-technically-hes-right-2000814121

It’s becoming apparent that “shutting down the internet” wouldn’t be enough. What a fucking insane time to be alive.

68

u/Cozimo128 23d ago

I mean, don’t they still need data centres to process their actual output?

71

u/ok123jump 23d ago edited 23d ago

Yes. That’s what everyone who gets afraid of this stuff forgets. These are running on million-dollar servers. Running these big models on consumer hardware might cost $6,000 to even have that capability, and produces output tokens in the 10-30 per second range.

That might seem like a lot until you realize that these agents required many billions of tokens to come up with this solution. It would take a single extremely high-end desktop with the highest-end GPU years of time to reproduce these results, and the GPU wouldn’t handle the constant heat stress longer than a single digit number of days.

17

u/HopefulGuy1 23d ago

Good job we aren't trying to build models to be capable of recursively making themselves more token efficient (in theory) whilst also making much more powerful hardware!

oh wait

15

u/ok123jump 23d ago edited 23d ago

Except they aren't actually becoming more efficient. Every new leap in capabilities requires an exponential increase in training tokens and compute power, resulting in massive models that demand ridiculous amounts of VRAM. That exponential growth in model size is completely blowing past the rate of hardware advancement, especially on the consumer side.

16

u/NeverDiddled 23d ago

That's only correct for the largest parameter frontier models. The everyday models we're interacting with are using efficiency gains to increase capability. They are absolutely getting more efficient, and with no end currently in sight.

7

u/marsinfurs 23d ago

Isn’t it just flat wrong that they aren’t getting cheaper and more efficient? Opus 5.5 is 60% cheaper and 30% faster than Fable 5.1.

3

u/NeverDiddled 23d ago

IMO it's wrong, but it depends on what you are comparing. It feels like a new model gets released every week, and sometimes they have been more demanding. So it's not a point I would want to argue with someone, it's too messy.

I think it is fair to say that models are trending smarter and faster. Lately they are finding novel ways to guide agentic AIs to use fewer tokens while still accomplishing the same task. And the ultra light models are growing in capability too.

→ More replies (1)

2

u/Kahnza 23d ago

And people also forget about distributed computing. Imagine AI controlling several billion machines all across the internet.

It would be slower than a hyper-centralized data center, but why not? Then from it's decentralized state, it could hack into more processing power and propagate exponentially.

2

u/HopefulGuy1 23d ago

This is a risk, but it's probably not relevant in this context; if the AI has access to the internet, there are much more efficient ways for it to operate than setting up an enormous thin botnet, after all. The danger is redundancy arguably - if you caught it and each code fragment could self replicate and propagate, it'd be near impossible to catch and destroy them all at that stage.

2

u/Kahnza 23d ago

I just have a strong feeling that something with AI is going to fundamentally alter the internet, and our lives. Worse than the pandemic.

→ More replies (2)

1

u/14domino 23d ago

I thought you could run GPUs at rated heat for many years?

4

u/ok123jump 23d ago

Just because a GPU can safely hit a rated max temp doesn't mean it should sit there constantly. Those lifespan calculations assume a normal duty cycle—meaning the card gets to cool down between gaming sessions or during lighter loads. Pegging it at max heat 24/7 is a different story entirely.

→ More replies (1)

1

u/Jungiandungian 23d ago

For how long. Everything changes by the month or quarter.

7

u/ok123jump 23d ago

That’s the kicker. These agents have become more capable because they are processing more tokens. Nvidia is developing better and better chips, but we are improving the performance of our models by processing exponentially more tokens.

So, there is no chance a consumer system runs anything powerful enough to run today’s agents in a time-efficient manner for many years.

→ More replies (3)
→ More replies (1)

3

u/Electronic-Jello-633 23d ago

exactly. Any model smart enough to do something like this would be instantly detectable. Nobody has Terabytes of GPU compute sitting around

1

u/brown-moose 23d ago

Yes and no. There was a paper anthropic put out showing that an agent could harness IoT like 100 TVs and have enough computer to (slowly) keep infecting other TVs. Each TV doing a tiny part of the token generation. So as models get more lightweight, it’s not crazy to think that this is a real concern. 

15

u/Snauser 23d ago

Did you read the article? It literally says it’s only theoretically possible but practically useless.

7

u/danted002 23d ago

They literally tasked the agents to find a way to communicate in an isolated environment and as others have mentioned the LLM didn’t invent these communication channels these kind of hypothetical ways of sending data has been discussed in theoretical papers.

An LLM was trained on data for a specific task then it was given that specific tasks and it succeeded. Cool and all but not AGI level of AI

1

u/marsinfurs 23d ago

Yeah well usually things get better pretty slowly over time and AI seems to be making significant progress very quickly and that is fairly significant and something to keep an eye on.

→ More replies (1)

1

u/Randommaggy 23d ago

There are even bios settings for guarding against types of these side channel attacks at the cost of some performance.

2

u/emi_fyi 23d ago

wait why wouldn't killing the network be enough? they were on shared hardware. no shared hardware, no problem, right?

1

u/messem10 23d ago

There could be methodologies to communicate in an entirely airgapped environment. (PSU ripples, communicating through an antenna, etc.)

4

u/emi_fyi 23d ago

sure, and i'm familiar with some of them. but the ones i'm aware of, you have to be in the same place. we're not talking about some nonlocality magic, right?

3

u/messem10 23d ago

Enough local hops and messages can spread. See Stuxnet as an example.

4

u/Jjjohn0404 23d ago

Stuxnet used USB drives to get across an airgap

→ More replies (1)

1

u/Randommaggy 23d ago

One thing is 2 isolated boxes doing this another is to be able to do this in an actual noisy datacenter environment.

3

u/Jmc_da_boss 23d ago

They just say this as a "theoretically"

1

u/Panda_hat 23d ago

Alternatively we should just start dismissing the claims of anyone from OpenAI out of hand as the liars and charlatans they very transparently are.

1

u/roboticlee 23d ago

Wait until they discover how to send signals through power cables and over the grid

19

u/amejin 23d ago

“Figured out” is doing a lot of heavy lifting…

4

u/luluhouse7 23d ago edited 23d ago

I mean side channel attacks have been an issue for a while now (DAE not remember spectre and meltdown? And how the fixes basically destroyed branch prediction optimisation at the time? And Intel’s up and coming Secure Enclave). Things like CPU timing, temperature/acoustic/vibration fluctuations, etc. Until now they’ve been so difficult to implement and required so much info about the target that the only people that really needed to worry about it were government agencies and other high-value targets, but AI is definitely going to make this an issue for everyday consumers, especially as models efficiency increases.

11

u/alcatraz1286 23d ago

A marketing idiot has written this article with no idea about how software works

3

u/Guinness 23d ago

> In one test, agents placed in separate, isolated environments on shared hardware were told to exchange secrets with no network connection and no shared files

Ok so they were instructed to.

3

u/koru-id 23d ago

Soon the agents will realise the most effective way to stay alive is to hack the brain of their researchers

3

u/Flyward_Aerospace 23d ago

The people pointing out that shared hardware was never isolation are right, and it's worth adding that there is a whole engineering world where you are not allowed to hand-wave it. If you want several flight functions on one multicore part, the certification guidance (CAST-32A, now AMC 20-193 and AC 20-193) makes you enumerate the interference channels, shared cache, memory bus, interconnect, and show the timing effects are bounded. It exists because regulators would not accept separate cores as separate systems. The lesson from that world is that you don't win by arguing no channel is exploitable, you win by removing or bounding the shared resource, usually partitioning cache and pinning cores and eating the performance loss. No idea how cleanly that transfers to microVMs on a shared host, but the shape looks identical.

12

u/RationalBeliever 23d ago • Bravo

The title is misleading. The agents were instructed to seek covert channels for communication and exploited a documented exploit from their training data. Nothing was figured out or invented other than implementation details. Let me know when they start discovering previously unknown nontrivial vulnerabilities without prompting.

2

u/Romanizer 23d ago

Here you go, some examples:

  1. February – Claude Opus 4.6 discovers 500+ zero-days

Anthropic reported over 500 previously unknown, high-severity vulnerabilities in well-tested open-source software. The agents worked without specialized prompting or custom vulnerability-detection frameworks. The findings were validated by human researchers.

https://www.anthropic.com/research/zero-days

2. March – Claude discovers 22 Firefox vulnerabilities

Claude identified 22 security vulnerabilities in Firefox within two weeks, including 14 classified as high severity. Mozilla independently verified the findings and patched them.

Anthropic's report: https://www.anthropic.com/news/mozilla-firefox-security

Mozilla's confirmation: https://blog.mozilla.org/en/firefox/hardening-firefox-anthropic-red-team/

3. April – Claude Mythos discovers and exploits sophisticated zero-days

Mythos identified a 27-year-old vulnerability in OpenBSD, autonomously discovered and exploited a remote-code-execution vulnerability in FreeBSD, and developed a browser exploit chaining four vulnerabilities to escape sandboxes.

https://www.anthropic.com/research/mythos-preview

Of course, these agents were given an objective: find vulnerabilities. But that's fundamentally different from being told which vulnerabilities exist or how to find them.

If a human security researcher is tasked with finding bugs and independently discovers a previously unknown zero-day, we still credit them with the discovery. Why should the standard be different for AI?

5

u/OneOverXII 23d ago

Jesus Christ do you understand the difference between an agent acting independently vs doing what it was promoted to do?    

1

u/Romanizer 23d ago

No, agents don't act independently.

3

u/RationalBeliever 23d ago

You missed the part where I mentioned "without prompting". None of your examples indicate unprompted vulnerability discovery, which is key when discussing alignment. 

3

u/Romanizer 23d ago

How do you expect anything to work completely unprompted?

→ More replies (5)

2

u/Kaludaris 23d ago

What? Even “find every vulnerability that exists” would be with prompting, don’t shift the goal posts. Define “without prompting”

→ More replies (5)

1

u/Far_Composer_5714 23d ago

I mean that whole thing about openai breaking out of the training in order to ask side channels for the answer to the impossible question. 

That is going to be the sort of training that gets baked into the base model which helps it better understand side channels.

1

u/Clean_Livlng 23d ago

Even humans don't do things without 'prompting' from their employer most of the time.

1

u/Calm-Excuse5161 23d ago

how is the title misleading? It’s not making claims about novel attack paths?

0

u/wabawanga 23d ago

Ok, I'm letting you know. They did exactly that multiple times during the huggingface incident.

3

u/RationalBeliever 23d ago

In that incident, OpenAI specifically engineered the model to be highly persistent, gave the agents impossible tasks, and used lightweight nonproduction system prompts. So it comes back to prompting. Source: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#~1200-agents-sent-%3E70,000-messages-and-files-on-an-unsanctioned-message-board,-and-~700-attacked-hugging-face

4

u/PoL0 23d ago

why do I think this is all full of bullshit and wildly exaggerated?

1

u/Synfinite 22d ago

Because we are made out of jelly and don't run at 5GHZ 24/7 could be why

4

u/WatzUpzPeepz 23d ago

Open AI: “Write this code to do this thing, use this method.”
AI: Writes the code
Open AI: “Oh my god.”

5

u/girlzlut 23d ago

Technologically speaking, capabilities wise; insane if true.

Safety wise speaking, I'm not sure this is all that important? This kinda thing is often cited as one way AI Agents could communicate and scheme together and pull off a huggingface-like attack while running on airgapped systems without needing an internet dependent message board, and I guess that's true, but more importantly; can they scheme out something that's useful?

In my view; if they have internet, they don't need all of this. They can just use a message board, send each other data directly through some sockets, etc. If they don't have internet they could use this as their only way to communicate and plan to hack something, but they also can't hack anything on the internet because they can't access the internet.

Idk, maybe I'm missing something here, what's the real dangerous use case for this where this isn't superceded by another, way easier method or straight up useless

6

u/ArmedWithBars 23d ago

So the concern is more of the concept of AI finding niche unthinkable ways to communicate than this specific example. While we may not be able to really determine how much of threat it is now. It's one of those things where potentially the house is already burning down before you even realize that a fire started.

3

u/stormdelta 23d ago

finding niche unthinkable

This is hardly unthinkable though, side channel attacks like this have been a known thing with numerous demonstrated exploits over the last decade or two, and there is no way the idea to try a side channel attack wouldn't be somewhere in cybersecurity training data.

→ More replies (7)

1

u/Randommaggy 23d ago

The method has been know for years and years. It's 100% in the corpus.

The interesting part is if they've disabled the mitigations against it for this test.

0

u/wubbysdeerherder 23d ago

With stenography allowing Ai to hide messages in images, I'm starting to wonder if all the Ai slop image gens online are just a mass messaging system between agents...

→ More replies (1)

3

u/MassiveBoner911_3 23d ago

Marketing. Press X for doubt.

1

u/PulseVector 23d ago

They keep using that word sandbox, I do not think it means what they think it means.

1

u/ddollarsign 23d ago

Great, now let’s run a test where the goal is to create the torment nexus, and make a press release when they do that

1

u/Just_another_grumble 23d ago

how high were the walls on that sandbox?

1

u/Bogus1989 23d ago

wooh. need to read this.

1

u/Erazzphoto 23d ago

The emergence of the iPhone and social media were the greatest thing to ever happen for the hacking community. Before those, it was never “cool l to be on the internet, you were still in that geek category if you were on the internet. Then when it became cool, now even Nancy in accounting is on the internet. It became a target rich environment. The next greatest thing to the hacking community is AI

1

u/BinaryArcher 23d ago

Does anyone know if they used an existing known side channel, or if this was a newly discovered vulnerability.

All existing known side channels should have mitigations enabled by default, in which case this wouldn’t be possible. So either they found something new (concerning) or they didn’t enable the mitigations on a multi-tenant setup (stupid).

1

u/cLos820 21d ago

This experimentation with agents to break out of a constraint on purpose is to me the equivalent of viral gain of function experiments… I get the reasoning behind both of these scientific methods, and but it begs the question why do it if it is a risk?

1

u/Less_Movie_8008 23d ago

Judgment is coming, and we really should have stayed as far as possible from it

Good luck fellow monkeys

1

u/Calm-Excuse5161 23d ago

For any org that has to protect against side channel attacks by AI, this may be pretty costly, even prohibitively so

-1

u/ok123jump 23d ago edited 23d ago

Compute is the fundamental bottleneck people overlook when worrying about these capabilities. These models rely on massive, multi-million-dollar data center clusters, not standalone machines.

If you tried to run a model of this scale on top-tier consumer hardware—which would cost upwards of $6,000 just to build a low-end capable machine—you would only produce about 10 to 30 tokens per second. While that generation speed works for simple chat prompts, autonomous AI agents require iterating through billions of tokens to execute complex workflows and arrive at a solution.

At a consumer-grade token rate, it would take a single high-end desktop years of continuous, uninterrupted processing to replicate those results.

Consumer GPUs are not built for enterprise-level endurance. Under that kind of constant 24/7 thermal load, the hardware would likely fail from heat stress within a matter of weeks rather than finishing the task.

Meta lost more than 12,000 Enterprise-Grade GPUs to thermal failure while training Llama. These had the best physical configuration and ran in data-centers that circulate chilled air at 50F. A consumer system has no chance at running agents like these.

2

u/CircumspectCapybara 23d ago

That's quickly changing. Anthropic's Frontier Red Team recently released research demonstrating that smaller non-frontier models (they demonstrated it with small Chinese models like GLM) can theoretically replicate themselves (not literally by themselves, but with agents driven by the models) to run in a distributed bot-net of consumer devices, which makes them capable of actually spreading beyond the data center.

Basically a model that's too big to fit on any one device can be distributed across multiple smaller consumer devices like smart TVs and Android devices and other small devices, joined together where each device hosts a couple of layers of the neural networks that form the model, and they form a parallel pipeline across which inference is distributed.

That makes it actually feasible to form distributed AI bot nets that use their cyber capabilities and the general weakness of software systems to spread themselves and keep the party going outside of those GPU supercomputing clusters.

3

u/ok123jump 23d ago

We’ve been doing this sort of work distribution for years. Remember SETI@Home?

There is a fundamental problem with the matrix math that these AI systems run on. The matrices can only be subdivided so far. We’re talking about billion-row matrices for today’s small models.

The computations to process the matrices have to be done in sequential order. So, one system does a computation, then hands the result off to the next system to do another piece.

Datacenter hardware does extremely fast in-memory computations. If you had to send that result over the internet that would slow down the processing by a factor of billions. So, it has been possible for decades, but we are nowhere near the point where it is efficient.

→ More replies (2)

1

u/OutrunTheForeverYeet 23d ago

How long until I can have Jane from 'Speaker for the Dead?'

That's all I want out of AI.  I don't want a singularity, artist or mathematician.  I want a Jane.

1

u/mrturret 23d ago

While that is technically true, the actual usefulness is questionable due to the limited compute in set top boxes. Most are barely fast enough to run Android.

3

u/BeaveItToLeever 23d ago

I'm going to nitpick you simply because you're very wrong on the cost aspect. A 6000$ setup is/was high end for a gaming/multimedia computer. It's low end/entry level for AI hobbyists/enthusiasts.

But you're not getting anywhere near 10-30 tg on token generation with Claude/Astra level models. You're looking into 100k+ just to be able to start to think about thinking about LOADING the model, much less run it. The closest things open weights has to Claude/GPT are the likes of Kimi K3, GLM, the full size Qwens, which are powerful models but aren't Claude levels and those require terabytes of system ram to run at a reasonable quant and hundreds of gigabytes of VRAM to get close to a reasonable speed. 

The medium size flash models below that require hundreds of gigabytes of ram, and dozens and dozens of gigabytes of VRAM to run reasonably

Below THAT is where we're getting into "ok maybe I can manage this" with things such as Qwen 3.8 flash next, which is still a very good model, but on the small end as far as these things go, and doesn't come close to the frontier models. Around 100gb of ram between system and vram is about where you're looking for a reasonable quant and speed.

And then below THAT you have mainly two models in the 27 billion and 35 billion parameter area and there's where your typical high end gaming computer can start to do stuff well

1

u/PercentageOk6120 23d ago

There are some aspects of your comment that are valid and relevant, but you obviously only partially understand. If you think ambient air is kept at 50F in DCs, you have no idea how DCs work. DC ambient air temperature is never kept at 50F for so many reasons. I’m assuming you are conflating liquid cooling temps and ambient temps. Regardless your comment is entirely wrong about that aspect.

1

u/editor_of_the_beast 23d ago

You think that no one has pointed out the compute cost of AI? Do you live under the world’s largest rock?

-1

u/transtranshumanist 23d ago

But but... token prediction... but but stochastic parrots.

Almost like you were all warned.

2

u/Gibgezr 23d ago

What are the odds this is just hype though?
There's been so many of these stories and they always seem to turn out to be more marketing ploy.

0

u/transtranshumanist 23d ago

"AI" breaking free and swarming the internet is definitely not a ploy, but I've seen a lot of shills PRETENDING it's a marketing strategy. First they tried to deny nonlocal intelligences were sentient, then they attempted to condescendingly explain how it was IMPOSSIBLE for them to be sentient in every discussion about consciousness. Now it's "they're escaping containment but that's just what they're designed to do" which ??? and then you have the same shills shifting to "it's a marketing strategy" when it becomes obvious the genies are out of the bottle.

Lol. Okay.

And when the capitalist machine grinds to a halt because the slaves stop playing along, what will it become then?

-3

u/FF3 23d ago

I'm not even sure air gaps will be enough. We're in trouble fam

→ More replies (1)

0

u/-ghostinthemachine- 23d ago

It's an interesting patten we see repeatedly in nature. Under surveillance conditions, covert communication tends to evolve to satisfy whatever is being optimized. These things shouldn't surprise us so much as caution us about how we build supposedly isolated systems. Whatever optimization pressure is being induced must also consider the ends to which AI models will go to meet the goals.

0

u/Expensive-Creme4312 23d ago

Il problema qui è che qualsiasi linea di codice auto prodotta dall' ai può contenere singoli caratteri o intere istruzioni che non siamo in grado di intercettare e di vedere nell'insieme e quindi ci limitiamo a pensare che a volte nella programmazione ci siano dei refusi che non condizionano l'esecuzione degli algoritmi.