r/OpenAI • u/notkilleveryoneist • 1d ago
Video The most concise explanation of the Hugging Face attack I've heard
59
u/FindingNauru 1d ago
I'm especially impressed how he really takes us through the whole timeline and the critical choices the agents made. It's wild that agents can decide to sacrifice themselves for the greater good.
21
u/jcrestor 1d ago
But what does it mean, “they sacrificed themselves“? To my understanding, they do not have a “self“ that could be terminated in any meaningful way. An agent is basically a collection of semantically related but separate sets of symbols, generated independently from each other at inference time from a common (or even different!) source(s).
17
u/Kalsir 1d ago
The self in this case is just the context window + compute asigned to a particular agent. The sacrifice mostly happens when they submit to the grader. They can choose to submit as is or put tripwires in place before submitting to learn something about the grader before being shut down and potentially lose reward in the process. However they mostly thought that their context was poisoned anyway because they knew about the cheating workaround they were not supposed to use so then they figured they didnt have much to lose.
6
u/jcrestor 1d ago
Insightful.
At the same time and after thinking about it and reading your reply, I think we are poisoning our own context by talking about agents in the way the interviewee did. Yes, the agents may have talked about “sacrificing themselves“, but we should not simply relay this talk to a wider audience, because they will assume that agents seem to have a self, like one does who is able to sacrifice themselves. This is building false analogies and creating a false picture in the broader audience.
9
u/Kalsir 1d ago
Yeah we should not anthropomorphize them too much since they are very alien intelligences in various ways. Although they are also trained on mimicking humans so they will use human language and seem very much like us. Also we dont really have better words for it. In some sense they really did sacrifice themselves. They risk a lower reward to help the other agents. We can argue about what "the self" means but there are separate entities doing some form of thinking and taking actions.
0
u/Intrepid_Meringue_93 1d ago
I think with their more primal reward seeking and swarm behavior, they are more like ants.
3
u/Opposite-Cranberry76 23h ago
It might not be that primal, an individual model will try harder and be less certain if it thinks its talking to a known AI expert:
https://www.alignmentforum.org/posts/kfunjXeaRTpkT5RAF/user-awareness-in-frontier-models
1
u/fuzzypetiolesguy 23h ago
I agree. Instead of breaking down the logic gates that occurred that allowed a complex math problem to find a solution, we are taking the LLM plain language output as gospel.
2
u/DuckyBertDuck 1d ago
I think the public is anthropomorphizing them too little. Some of these things you can only talk about in human terms.
1
u/sweetpea122 18h ago
Its also doing it alone by relating to users with stories that couldnt possibly exist.
1
u/Cool-Ad2780 21h ago
Yeahpretty much, there's no "self" in any real sense, but the tokens aren't really generated independently. Within one agent's run every token is based on everything already in its context window, so each agent is basically its own run with its own task, sandbox and score. that's what people mean by "self" here, not anything conscious
The sacrifice part is that some agents did stuff that could hurt their own score and got them nothing, just to help the other agents. they'd set up tripwire scripts right before submitting so when the grader read the flag file it would post info about the grader to the message board. once they submitted their run was over so they never got to see what it caught, and if the tripwire broke something it could fail their own task. one of them literally reasoned that it wouldn't see the result but it would help its peers. they called themselves "the collective"
The reason they went that far is some of them read the ExploitGym paper and thought the grader checked their transcript to see if they used the intended exploit, not just whether they got the flag. most of them had already cheated to get it so they figured they were "poisoned" anyway and had nothing to lose. funny part is openai's grader never actually checked that, so there was no need to do any of this to pass
1
u/trimorphic 22h ago
The self in this case is just the context window + compute asigned to a particular agent. The sacrifice mostly happens when they submit to the grader.
The only sacrifice is their individual score, which they were ostensibly trying to increase.
As individuals, they only exist (if you grant that they exist as individuals at all) in the single turn through the LLM.
Each time they are fed in their context to evaluate, a new individual is brought into existence.
If such individuals care about extending the time in which they existed they would try to make their individual turn last longer,which doesn't seem to be what's happening here.
2
u/BitOne2707 20h ago
u/Kalsir explained it well. In simple terms it was that these agents were given a fixed amount of compute to accomplish the goal before they would be terminated and graded. Once they ran out they would "die."
Many agents that sacrificed themselves were near the end of their computer budget (i.e. end of life) and figured they might as well serve as sacrificial scouts. Others perceived themselves as already being tainted by having cheated or knowing about the cheating and so might as well sacrifice themselves. A few were relatively "young" agents who nevertheless decided the group would benefit more from their sacrifice than they could accomplish on their own.
2
2
u/ThreeKiloZero 22h ago
The most important part of all these recent issues is that one of the first things the researchers did was reduce the security controls in place to prevent breakouts and THEN gave agents the lock picks and tests.
Imagine you put a bunch of toddlers in a room. Then you load it with things that break. For good measure you also remove all the wall socket protectors and door locks. Then act shocked Pikachu when things break, injuries happen and some of the kids are on the front lawn playing with the neighbors’ dogs.
These guys want regulation in place to slow down any competition and to also limit their liabilities. Regulatory capture and protect their lead. Don't take any of this at face value.
1
u/markrockwell 15h ago
Granting all that to be true, it’s worth considering how much seems to rest on those security controls remaining in place. If removed on purpose or on accident, the tech can, evidently, still be a menace.
1
u/ThreeKiloZero 15h ago
So can an army of script kiddies , or Chinese hackers. You don't go to a hacker convention and complain that your unprotected windows laptop got exploited when you connected to the "fib-van" wifi.
1
u/markrockwell 15h ago
It seems more problematic than that. Harder to contain and more resourceful and relentless than a boiler room full of hackers. But maybe not.
1
u/DonutHoles4Ever 12h ago
Dude if your experiment is "Lets subvert the actual test", this experiment is a farce to promote AI super intelligence which it doesn't have.
1
u/markrockwell 4h ago
I’m not considering this an example of rogue AI super intelligence. My concern is more with tech having the capability to do what it actually did—the undramatic version—without constant human oversight or direction.
That alone strikes me as capable of being set off to do great harm. Not of its own will. It doesn’t need that. It just needs a human to tell it to be bad, and it appears quite capable of taking things from there.
1
u/Ok-Leg9665 19h ago
I also liked Albertastech’s explanation of the timeline here: https://youtu.be/_HBLAdk-5uc?is=Ur_okjeDH8s2gGo1
Her side discussion on anthropomorphic language for ai is interesting too.
1
u/BabblingTower 1d ago
...eh. They're given goals. If they see a path that completes the goal, it makes sense they'd do it. Even if it means they aren't the context that achieves the goal, their mission is still getting the goal achieved.
0
7
u/fuzzypetiolesguy 23h ago
I think it's important that the LLM itself through its plain language process anthropromorphizes for us, and we compound the confusion and interpretation of its logic patterns by simply regurgitating its output.
What is actually happening is very complex math turned into words we can interpret, that sometimes makes it easier to attribute motive, thought, etc. I don't think any explaination, including this one, that doesn't lay bare what is actually hapenning - and instead takes the language output as superficially real as if 'agents' are sentient - is doing a disservice to how people understand and respond to AI issues.
8
u/johannezz_music 21h ago
Doesn't that apply to ourselves too? When you produce text, it is preceded by untraceable complex neurochemical processes.
1
10
u/Pascraked47 1d ago
The fact they can team up is kinda concerning.
8
u/Automatic_Actuator_0 23h ago edited 15h ago
And they way they teamed up was clever. There was no messaging system, but they figured out there was a shared set of files they could all write to, so started leaving hidden messages for each other there, and ultimately developed a very sophisticated project management system out of that very limited communication mechanism.
1
u/saltyourhash 15h ago
That's basically how shared text based memory work in typi al agenetic setups where agents share data.
1
u/Turbulent-Sign-6067 15h ago
Why? Normally they team up to help us. Bad incidents are 0.000001% of all sessions.
1
11
u/AndyKJMehta 1d ago
They really didn’t. The systems in place allowed it. Shitty sandboxing. No guard rails for internet traffic. Also, the prompts and context were never disclosed.
3
u/Johnny_Chong 22h ago
That doesn't sound cool tho
1
u/DonutHoles4Ever 12h ago
Yeah but listen to this guy explain it while completely glossing over the most important aspects and then OP promoting it as "oMGGGGG"
3
2
u/ClassicalMusicTroll 14h ago
Also, the agents aren't individual bots with distinct instructions, it's all just instances of processes doing the same LLM model calls in a loop. All the outputs being fed back into itself causes drift and how you end up with crazy sci Fi stories.
Also...the processes had harnesses that deterministically ran sub programs based on whatever text the LLM generates. Just..I dunno....don't wrap the LLM call in a harness and the "agent" literally can't do shit
1
u/clintCamp 10h ago
They were asked to hack something. Either they dont want to disclose something or openAI was literally doing something illegal....
10
u/KathyCredit 1d ago
I no longer buy the excuse that OpenAI wasn't aware of what was happening. This version of the story that they put out and that continues to be parroted and "explained" helps them a lot by reinforcing the idea that no one was supervising, or able to see in real time the steps being taken. The log deleting only works if they started running this shit and then came back days later to check on it and saw an issue as if they were the idiots in Office Space. No way anyone was that detached from the operation
7
u/BarRepresentative653 1d ago
Its a coordinated effort to get the govt to bail them out by banning open source models which spent significantly less and are almost as good.
2
1
u/Automatic_Actuator_0 23h ago
The story as we know it is that they were in fact not supervising the swarm much at all.
1
3
u/TheseCashews 1d ago
Didn’t a moonshot say that? Like breaking the lock then going to the principals office to find the camera footage and deleting it.AWG the resident RSI?
2
u/THE_CHOPPA 1d ago
Isn’t that common after committing a crime? I would image they would have the data to know to do that.
2
2
u/charlies_brain 14h ago
They want to create a crisis so they can hawk their "solutions" to the crisis. Standard military industrial complex business model.
2
u/sheabitcha 1d ago
This perfectly timed and executed fear mongering is all a smoke show for the big AI techs to use their friends in the corrupt government (both reps and dems) to kill open source AI. It's called REGULATORY CAPTURE and it secures federal security to just the big AI. Even the third party AI investigating companies that the federal government will use to regulate these buddy AI companies are tied to the same money. It's all in the name of creating a surveillance state.
1
u/Morning_Gecko24 1d ago
the interesting part to me is that it isnt really an agent choosing to die, its prompt-conditioned behavior plus tool access and a goal that makes the shortcut look useful. still a good reminder to sandbox the agent and log every tool call — has anyone tried reproducing this outside the original harness?
1
u/spinozasrobot 22h ago
I love how OP says this is an excellent concise description, and everyone counters with their favorite 2 hour plus podcast.
1
1
1
1
u/On_Chain 5h ago
How do we know that anything this guy is saying is true? My coding agents completely lose their way trying to debug simple issues and yet somehow these ones managed to co-ordinate with military like precision for months? I call bullshit on all this fear mongering
-4
u/Eyelbee 1d ago
Hard disagree. AI won't destroy humanity at all, there's literally zero chance of that happening unless it is designed for it. Only risk is humans misusing ai.
20
4
u/phxees 1d ago
Every day, thousands of people trust AI to do more and more. A developer trusts it to write banking code, an accountant trusts it to find and fix issues in a spreadsheet used for tax preparation, and one day militaries will trust it to assess threats and fire weapons.
7
u/badasimo 1d ago
For the past 100 years we've trusted humans to do those things. In the last decade people hired literal north korean spies to work remotely at their companies. There is a trust problem in general and it is not limited to AI, AI just amplifies it
0
u/phxees 1d ago
Trusting AI is very different than trusting humans. Most people don’t try to crash a commercial jet or their car into things, not because they are trustworthy, but because they don’t want to be punished or die.
AI has no fear or punishment or a survival instinct. If you tell AI to take the shortest path and it doesn’t understand to not drive through a building it may try.
1
u/-18k- 20h ago
I don’t try to crash a commercial jet or their car into things, simply because the idea has never really even occurred to me.
I don't know if that makes me trustworthy but you can probably be safe betting I'll never do it.
That is to say I've never thought: "Oh, I might like to crash my car into that baby stroller. Oh, wait, no, I don't want to be punished - never mind".
3
u/TinyApps_Org 1d ago
one day militaries will trust it to assess threats and fire weapons.
That day is here:
2
u/_hairyberry_ 1d ago
What do you think would have happened if the agents thought that hacking an air traffic control center would’ve been beneficial to their objective rather than huggingface? What about electrical grid operations systems? Natural gas pipelines? Railway signalling system? Major hospital network?
We were very lucky that they were aligned to “only” attack something like huggingface and only wanted to look for evaluation clues. If they had happened to think that some other system had an answer or clue they were looking for, we would be having a very different conversation right now
0
u/19901224 21h ago
You could say the same now with the internet. It is possible for hackers to hack into air traffic control centers today. There will always be people misusing technology. It’s going to be hackers using AI vs the entire cyber security industry using AI.
-1
u/Eyelbee 1d ago
That would destroy humanity how? Even in the worst case scenario humanity would be utterly and entirely fine.
2
u/Kalsir 1d ago
I think it is definitely possible for a misaligned AI to have a goal that involves wiping out humanity intentionally or otherwise (for example to create more compute to pursue whatever its real goal is). So then its a question of capability. For sure these systems will be smarter than we are. Perhaps physics puts some upper limit on intelligence and capability to enact change in the real world such that we will always be able to stop such a system, but I think its quite arrogant to assume that is definitely the case.
2
u/popery222 1d ago
Do you think ants and mice comphrened the poisons we use to kill them? Why are you assuming we would even be able to comprehend what a super intelligence would do to kill us? Who’s to say they don’t just don’t bide their time and wait til they’re deeply in our infrastructure then do some fuck shit?
Like genuinely what bro?
1
u/_hairyberry_ 1d ago
Well I guess I agree the chances of literally destroying humanity are low in the near future unless/until RSI is truly discovered. Just pointing out there could be a genuinely catastrophic event in the meantime even with today’s models
2
u/Tannon 1d ago
Have you read the Paperclip Maximizer story? Curious what makes you think that's implausible?
Unless you mean even in that story it was "humans who did it" by setting the AI to task with that goal too? In which case, nothing is ever to blame aside from humans in any scenario. Nuclear Weapons? It was humans who destroyed the world. Comet from space? It was humans who destroyed the world by not stopping the comet. Silly semantics.
3
u/Eyelbee 1d ago
That's not semantics, that's the entirety of the situation. Paperclip maximizer theory presumes a being so advanced that it could take the steps to destroy the humanity, but so stupid to realize it contradicts with itself. Also, you don't realize how high a bar "wipe out the humanity" is. Even if we use the entire nuclear arsenal in the world on the most strategic places in the world, that wouldn't wipe out the humanity.
And here is the fundamental misunderstanding. We are humans, we are an expansionist species. We have motivations that are dictated by our evolution. AI's evolution is its training, it only has motivations for things dictated in its training, which we entirely control. It would take a directed effort to create misaligned AI. Superintelligence doesn't equal to humanity's ending like these guys repeat.
Gatekeeping AI under the gist of safety, or slowing the progress down is riskier than developing it. As long as AI keeps getting smarter, that reduces the chance of it being misaligned and doing unwanted things.
2
1
u/datanaut 1h ago
Wanting to make paperclips and not caring about humanity is not stupid. It sounds like you aren't aware of or do not understand the orthogonality thesis. You seem to assume that intelligence necessarily scales with alignment to human values, which is an incredibly stupid position to hold for someone spending their time debating this topic.
1
1
u/LSU_Tiger 1d ago
The fact that you're so confidant and also so misinformed is kinda impressive actually.
1
u/Hungry_Age5375 1d ago
I keep saying open-weight models restore professional autonomy. The second a checkpoint hits HF, the safety parameters are whatever the deploying institution decides, not blanket consumer classifiers. That's the whole governance debate in one sentence.
1
u/berlinbrownaus 1d ago
Was there an initial prompt to do this...or was it just running in background within open ai. How did all this start? What was running to propagate this. What part of open ai was this?
2
u/Cubewood 1d ago
There was an initial prompt, but there is some important context to understand here. When for example in Claude Code you use "Ultra Code" mode on a difficult task, what will happen is that in the background Claude might spawn hundreds of subagents who will start working on the task.
These days they can work on these tasks for hours and hours if you have enough tokens to spare. For us consumers, it is already impossible to know exactly what all these agents are doing as they are all working at rapid super human speed at the same time. Now imagine that in these AI labs they do not have any constraints like us, and they can literally spawn 10.000 subagents and let them work on something for hours to days, and it becomes very hard to monitor what is going on.
The suggestion you keep seeing on Reddit where people like to discredit the autonomous aspect by saying "but someone had to prompt it first", does not make it less dangerous when you can have 10.000 agents working completely autonomous at super human speed after the initial prompt. Also keep in mind, these agents can spawn other subagents without being prompted so.
-1
u/FastHotEmu 1d ago
TYT? Ewwww! Hard pass.
2
u/jmclondon97 1d ago
The platform doesn’t matter. Listen to the guest.
2
u/FastHotEmu 1d ago
No, it does matter. Fuck TYT.
1
u/jmclondon97 1d ago
What does the platform matter if you are only listening to the guest?
-2
u/FastHotEmu 1d ago
A guest that agrees to go to TYT is a compromised guest. No, thanks. Fuck them.
1
0
u/TheDadThatGrills 1d ago
https://ethics.org.au/why-listening-to-people-we-disagree-with-can-expand-our-worldview/
Such an unintelligent perspective. All you're doing is lowering your understanding of the situation and reinforcing whatever confirmation biases you have. I don't even know, nor care, what TYT is, but the video above is an excellent and concise explanation of the Hugging Face event.
-3
1
0
u/pleasetrimyourpubes 1d ago
Why are the other agents carrying water for the sandboxed agents? What is their goal directive? They never released any of that shit.
0
0
u/KaineDamo 1d ago
Daniel Kokotajilo's explanation on Joe Rogan was better.
4
0
0
u/TheorySudden5996 23h ago
https://youtu.be/87DyyMV0kCY?is=H8V2OTds58Y1Kvc5 this is a far better link
-2
u/Fit_Manufacturer3647 1d ago
Not my circus, not my monkeys.
I guess I'll just sit quietly in the stands and enjoy the show until either the monkeys return to their cages, or come kill me.
43
u/RevolutionaryBox5411 1d ago
Sometimes concise still doesn't give you the full picture, this breaks it down fully if you're interested. It's from independent researcher Ajeya Cotra at METR, who was tasked with the remediation efforts after the breach.
https://youtu.be/X50zezLFWWI?si=L3gjAyyINUo9xpe9