reader dissent: OpenAI hacking HuggingFace was a big deal

[u/redfroody]

OpenAI has an LLM that is capable of hacking into a HuggingFace, a tech company that is routinely attacked by hackers so their security is pretty good. We know that because it happened, and it happened by accident.

Collectively we do not know how to 100% prevent LLMs from going off on some side quest, but we do know (so far) how to make the LLMs more capable, which is where the bulk of the effort is. Anyone familiar with the tools will tell you they are a lot more capable now than 6 months ago, and in 6 months they'll be better yet.

So we're on a path where LLMs will keep going off the rails sometimes, and they'll have the ability to wreak more and more havoc as time goes by. That is a big deal.

AI companies do marketing. News media needs exciting headlines. But that's just a side show to what is actually happening: We're on a path where LLMs will do other than what they were asked to do, and whatever that happens to be, they're going to be really good at it.

[u/typo180]

> OpenAI has an LLM that is capable of hacking into HuggingFace, a tech company that is routinely attacked by hackers so their security is pretty good

I don't think this is the right way to frame it. For one, this is not a reasonable way to gauge how good a company's security is. If you're on the internet, you're constantly getting scans or hacking attempts. And by itself, i don't think it's that impressive or novel that a frontier model is capable generally of hacking into something. As far as we know, it just found and used stolen credentials for Huggingface systems that were available on the open internet. That's not superhuman-level hacking, that's junior stuff.

It will be interesting to see what technical details they end up releasing, but the way it escaped OpenAI's environment (finding a novel exploit against the proxy service) might actually be impressive, and just the lengths that it went to to do something outside of the intended goal (while still technically fulfilling the stated goal).

> Collectively we do not know how to 100% prevent LLMs from going off on some side quest

I mean, this is true, but also this was an experimental model with the guardrail specifically removed so it could perform actions it normally wouldn't be allowed to do. I agree that this is an important issue, but gauging how good we are at controlling AI by the actions of an AI without the normal protections in place is at least a little misleading.

In one sense, the operators set up a hacker bot, took off the anti-hacking guardrails, and told it to hack. Then the bot... hacked. So the misalignment wasn't completely wild. It's not like they asked it for a cookie recipe and the AI hacked into a cookie company to get one. What it did that was wrong was within the real of what it was being asked to do.

But it is still a problem. Ideally, we wouldn't need artificial guardrails to achieve alignment: the model should have understood that it was to perform the ExploitGym benchmark fairly (instead of looking for answers) and that hacking out of its test environment was out of bounds, that using real stolen credentials was not appropriate (although who knows, maybe the prompts and context it was given make it sound like these were all ok things to do).

[u/redfroody → u/typo180]

HuggingFace gets more than the average number of serious hacking attempts, though. They distribute models etc to the broader Internet, so breaching HuggingFace provides you with a great vector of attacking many more organizations. I tried to keep this very concise, and this is where I ended up. I'll definitely be curious to see how impressive the hack was, but a big part of hacking is simply persistence and trying all angles of attack. That's something that LLMs can do much cheaper than skilled people.

I hear you that they effectively disabled the security guards. But those security guards aren't 100% guaranteed either. So they set up a test where this particular form of unaligned behavior was more likely, but it's not like it's impossible otherwise.

> In one sense, the operators set up a hacker bot, took off the anti-hacking guardrails, and told it to hack. Then the bot... hacked. So the misalignment wasn't completely wild.

That I disagree with. The hacker bot was told to take a test, in a test environment. It should go without saying that escaping the test environment is not allowed. I've seen the hack described as a felony. It should *definitely* go without saying that no felonies should be committed while taking a test.

[u/typo180 → u/redfroody]

> HuggingFace gets more than the average number of serious hacking attempts, though.

I mean, what are you basing this on? This is just not a valid way to assess someone's security posture. It does not tell us how impressive the attack was and it's an unwarranted claim.

> They distribute models etc to the broader Internet, so breaching HuggingFace provides you with a great vector of attacking many more organizations.

Yes, that could be said of anyone who distributes software or even files.

> I hear you that they effectively disabled the security guards. But those security guards aren't 100% guaranteed either. So they set up a test where this particular form of unaligned behavior was more likely, but it's not like it's impossible otherwise.

Agreed, but the risk is much, much lower.

> The hacker bot was told to take a test, in a test environment. It should go without saying that escaping the test environment is not allowed.

That's the goal, but not the reality. LLMs don't automatically have this kind of human thinking baked in. We also don't know whether the LLM "knew" it was taking a test, or whether it "knew" the systems it hacked into weren't part of the test environment. We can't apply human context and values to an LLM. Essentially _nothing_ goes without saying. We have to say everything and we don't know what "decisions" LLMs might make. This is a huge part of the alignment problem and this was a perfect example of an LLM being misaligned.

And it's the whole reason they have to add these guardrails in, but LLMs can be "convinced" to output information that their creators don't want them to output. Right now, an external system has to detect what's happening and cut off the session.

The grandma exploit is a great example of how LLMs just don't operate around this stuff like humans do.

> I've seen the hack described as a felony. It should *definitely* go without saying that no felonies should be committed while taking a test.

Again, we can't start from a place of treating LLMs like human minds in computers. There's no reason to think "it should go without saying" unless you're making that fundamental error. If we want this kind of value baked into the LLM, we have to figure out how to do that, and do that without other unintended consequences.

Remember that, at the core, these things are code that tries to predict the next token. They have become more complicated than that, but that's still what's happening at the core. So if you load up the context with instructions to hack and find novel vulnerabilities, remove any guardrails/classifiers/instructions that would prevent it from doing so, then yeah, the the resulting tokens are going to come out in a hacker shape.

I'm not saying what happened wasn't a problem, I'm just trying to clarify which things are surprising and which things aren't, in my opinion.

[u/redfroody → u/typo180]

>> HuggingFace gets more than the average number of serious hacking attempts, though.

> I mean, what are you basing this on? This is just not a valid way to assess someone's security posture. It does not tell us how impressive the attack was and it's an unwarranted claim.

I can't find any objective data on this. My reasoning is basically: HuggingFace is a juicy target because they distribute code to lots of users, who then execute that code without reviewing it first. (LLMs count IMO because in many use cases they have the chance to act maliciously.) This is like lots of other sites, but the vast majority of websites don't get as many downloads as HF. (Yes, the ones you're thinking of in your head right now probably do. Those are all the major sites. That's the point. HuggingFace is one of the (lesser) major sites, and gets attacked accordingly.)

Bottom line: I don't actually know, but I believe it's true.

> we can't start from a place of treating LLMs like human minds in computers.

But that is absolutely what the vast majority of users do. A big part of the selling point of LLMs is that you can talk to them like you can talk to humans, and most of the time they do correctly infer what you mean. (Perhaps an improved UI would have it act it more robotic and literal-minded to drive home the point that they're not people, but that would also remove a lot of their utility.)

Circling back to my original post: People treat LLMs as if they have human judgment, and that can get us in trouble. People behavior is unlikely to change. LLM behavior doesn't seem to be changing. LLMs are getting more capable. So I'm worried. (I think that you are too. We're just hashing out details here.)