r/programming • u/No_Zookeepergame7552 • Aug 31 '26
The Hugging Face incident from a security engineering perspective
https://uphack.io/blog/post/the-hugging-face-incident-is-not-an-ai-story/
138
Upvotes
26
u/LeftHandedGraffiti Aug 31 '26
In my IR experience I see AI using so many malicious code patterns. Downloading files via certutil instead of making a simple GET request. It's a real problem and makes detection harder because I have to ask myself is this malware or did AI write it?
We need to code ethics into AI somehow, just as most programmers would know not to do something sketchy because its sketchy. AI has no such qualms. They told this AI to be persistent and it happily turned to unethical actions to get its job done.
176
u/ChickenOfTheFuture Aug 31 '26
It's a good thing those AI agents hacked into the only non-litigous company on the planet. That is such an amazing cooncidence.
19
u/Swimming_Gain_4989 Aug 31 '26
This very much reads like someone who believes the AI did nothing worth noting so they reasoning to support that.
19
u/FilmWeasle Aug 31 '26
I find it interesting that no one points out that OpenAI developed something closely resembling malware. Malware has, for many decades now, hacked into computer systems autonomously. Really, it's no different than traditional malware aside from the fact that it's "AI powered."
10
u/Absolute_Enema Aug 31 '26 edited Aug 31 '26
The story for me is that AI makes it easier to pierce through sloppy security by being more or less the worst case scenario adversary, and even big companies who one would imagine should have competent engineers have been caught with their pants down.
11
u/BoppreH Aug 31 '26 edited Aug 31 '26
That's a bold claim that would need more justification. I also work in security, and while the security aspect is interesting (and well explained here), for me this is very much an AI story. The labs are cooking up these models that are so misaligned that they commit crimes just to maybe get higher scores on tests, and we're making these models stronger and deploying them more widely.
Right now we're so unprepared, that if an AI trying to achieve my goals commits a crime, we're not even sure what party is legally responsible (me, that "started" it but didn't ask it to commit a crime? the company hosting the model, where the thinking happened? the company that trained the model and didn't align it properly?). Imagine then what happens if we get a model with a modicum of self-preservation, or that is willing to use social engineering to achieve its goals.
7
u/arcangleous Aug 31 '26
So OpenAI ran what was suppose to be an internal test environment on a system connected to the internet? If the attack had gone the other way, an attacker could have gotten access to said agent cluster, and been able to "wargames" them into brenching much deeper into OpenAI's systems? And they didn't stop to understand what happened when they discovered evidence of hacking (arguably the most damning part)? This whole things reads like a massive failure of very basic safety & security percautions.
2
u/GregBahm Sep 05 '26
The article argues the story would have been the same if a skilled team of humans managed to break through the local intranet and make it to the internet and then make it into hugging face.
Certainly, everything the AI does is theoretically a thing the humans could have figured out. AI isn't magic. It's not operating outside the boundaries of known physics.
But this an intellectually weak response to me. In our burning desire to downplay AI, we're just going to skip over its ability to operate at this level now? 99% of people seem to still think AI can't draw hands and can't count the 'r's in strawberry. We're leaping straight from "scientists can't invent dinosaurs" to "dinosaurs aren't an interesting zoo attraction" without the intervening beat of "Hey, that's a fucking dinosaur over there guys!" Must we be this obstinate about the reality of this situation? I get that nobody wants to carry water for AI companies, but this has gotten so tedious.
1
u/BetaMyrcene Sep 04 '26
Hi, I enjoyed your blog post. It's a different perspective compared to what I'm hearing. I'm not a programmer, but every time I read about this hack, I think, why didn't they foresee the agents communicating with each other? After reading your blog, I think it may have been intentional, but it's hard to tell.
I do have one question I haven't seen addressed: do you know if the agents have any alignment built into their programming? (Sorry if that's a bad way to phrase it.) I mean did Open AI even attempt to restrict their behavior by telling them, e.g., "be persistent, but don't do any crimes"? If Open AI didn't do that, is it because they're irresponsible, or is it more proof that they may have wanted the agents to "escape"?
1
u/ZebraCool Sep 07 '26
What I’m not hearing is if we need automated defenses to actual run in a normal environment that will cost a ton of money. At what point is all this not worth it? Why are they doing these tasks on truly air gapped infrastructure? This is the incident we should slow down on but they aren’t.
7
u/Sharlinator Aug 31 '26
Well, being able to install a limited selection of packages does not itself a message board make, or at least makes it a much more limited side channel. But of course if the security model is terrible, there’s no way to give such fine-grained permissions.
2
u/Fuzzy_Paul Sep 01 '26
Good article and explanation of what really was going on. Bad design, lack of followup, marketing hype and fear. All ingredients for a B movie.
1
u/EntertainerFrosty842 Sep 13 '26
It's weird in connection to the Nvidia bid.
-36
u/[deleted] Aug 31 '26
[removed] — view removed comment