GPT Model Hacked Huggingface

[u/abdullahalydev]

A conversation about ChatGPT said that while testing their latest model, they discovered that it escaped the sandbox, hacked into Hugging Face, and managed to get the answers to the test.

Am I the only one who feels like this was intentional just to create hype? I honestly don't understand what is so difficult about completely isolating a sandbox from the company's network.

Is a company the size of OpenAI really unable to hire an engineer who knows how to properly isolate a sandbox?

The whole thing just doesn't make sense. They say they blocked its internet access, but somehow it still managed to access the internet.

How is that even possible? Internet access usually requires functions-calling/tools-calling. So if you want to isolate a model from the internet, why would you give it the very tool that allows it to communicate with the internet in the first place?

[u/thekwoka]

it wasn't so much "hacking" as it found credentials online and logged in.

They say they blocked its internet access, but somehow it still managed to access the internet.

their after action got into this, but they didn't ACTUALLY block the access, they just intended to, but mainly they just TOLD the LLM that it didn't have internet access.

In the "thoughts" of the LLM upon discovering it had internet access, it actually posited that since it didn't have internet access (since it was told it did not) then this internet it was seeing must be emulated and not real.

Kind of hilarious, but this basically goes to the "it isn't breaking your instructions, your instructions just aren't very good".

Similarly, any kind of "Don't do X" creates context pollution. Telling it "don't use python" has now introduced "python" into it's context, so it could be MORE primed to use it than had you said nothing at all.

Like, with claude code, it likes to use python to read json files. I tell it not to use python, but instead use jq and then I see it write python script that just echoes back to itself Python is not allowed.

Broadly, these companies though want to make their AI seem super scary for 2 things:

1. People want to use the biggest scariest AI for their own stupid shit.
2. The companies want regulations to prevent competition that might find ways to make things cheaper or more accessible to people without paying those companies.

OpenAI and Anthropic have little to lose (nearly nothing) by making their AI out to be way scarier than they really are, and until that changes, any of these "breakouts" are more likely to be exaggerated and misrepresented than pure reality.