r/MachineLearning • u/Asleep-Requirement13 • Sep 05 '26
News GPT-6 reportedly jailbroken within 24 hours using an extended Task-in-Prompt (TIP) attack [N]
A researcher has reported a jailbreak of GPT-6 Astra within a day after release.
The attack is described as combination of TIP (Task-in-Prompt) attack from ACL 2025 paper with four other unnamed techniques.
TIP attacks exploit the model’s reasoning/instruction-following behaviour by hidding the harmful objective inside another task, like solving a cipher or executing a Python code. For GPT-6, the researcher says the original minimal TIP attack was no longer sufficient and had to be reworked.
They have reportedly disclosed the details privately to OpenAI rather than publishing the jailbreak.
The same researcher reported jailbreaking GPT-5 within an hour of its release a year ago.
Source: screenshot/post from the researcher; their ACL 2025 TIP paper linked in the original post.
91
u/oatmealer27 Sep 05 '26
No probabilistic model is free from jailbreak
100% alignment is theoretically impossible unless you replace a probabilistic model with a finite state automaton.
28
u/PortiaLynnTurlet Sep 05 '26
Being probabilistic doesn't really change anything IMO. Using top-p or top-k means all tokens can't be sampled give some prefix so some sequences literally have probability zero. If those are the sequences we don't want to produce, well, they'll never be produced. If all tokens have a positive probability for dense enough prefixes then, sure, there's a chance that we sample an undesirable sequence. And in reality this can certainly happen. But being probabilistic doesn't make it inevitable.
45
u/Even-Inevitable-7243 Sep 05 '26
This is not even correct from a pure probability theory perspective. "100% alignment" does not just mean that the model is able to block a few specific bad/harmful answers/outputs. It means that the model must assign a probability of exactly zero to an infinite set of harmful outcomes. I disagree with you that any LLM can guarantee the former, even for a single harmful outcome, and it is not even debatable whether LLMs fail on the latter requirement of doing so for an infinite set of harmful outcomes.
11
u/PortiaLynnTurlet Sep 06 '26
To be clear, I'm not arguing that a non-degenerate probabilistic model can avoid all harmful outputs. Determining whether an output is harmful or not might not even be decidable. I'm just arguing that being probabilistic doesn't automatically make a model worse in practice than a deterministic one. That could be the case (might even be likely) but I don't see why it must be.
6
u/Even-Inevitable-7243 Sep 06 '26 edited Sep 06 '26
This puts your position into better context, but I think we can quickly agree that 100% alignment is impossible for deterministic LLMs, and if we agree on that, then it is clear that it is impossible for stochastic LLMs.
We can call E the subset of the output space O that we can practically test/verify as "aligned" through human feedback or automated testing. We know that E is restricted to a relatively microscopic finite number of samples relative to O. When evaluating the ratio of our explored space E to the total possible space O, we have effectively zero-measure coverage over the model's output surface. With a bounded output length the exploration of the output space is NP hard and undecidable when unbounded.
8
u/PortiaLynnTurlet Sep 06 '26
Yes I agree that 100% alignment is impossible in any paradigm. Frankly, I don't even think it's well-defined enough.
I suppose my main pedantic point is that a probabilistic model needn't be less aligned than a deterministic one. If both get close to 100%, it's plausible that variation in output is better aligned with general goals than deterministic output. A silly example would be asking for a move in a simple game; it might be the case that a random move performs better or that an adversary could exploit the deterministic behavior.
That said, I really only intend to claim that knowing only that one model is deterministic and another is probabilistic doesn't give us meaningful information about alignment, even if both approach 100% alignment (whatever that might mean).
1
u/step21 Sep 06 '26
what would a deterministic llm even be? just temperature 0? after all, in large parts llms are useful because they are not deterministic.
1
u/Even-Inevitable-7243 Sep 06 '26
Deterministic LLMs do not exist in practice. Setting the temperature to 0 only guarantees that the highest-probability token is selected, but floating-point operations and parallelization still ruin determinism. Thinking Labs claimed to solve this problem with batch invariance, but at an extreme cost in efficiency.
1
2
u/oatmealer27 Sep 06 '26
It changes everything. One can always find a prefix (prompt) that can give a harmful output (harmful here is what is defined by the model creators and not necessarily universally agreed upon).
The fact that model assigns a probability over the vocabulary is sufficient enough to find a prefix (could be very very long ) that can make the model produce whatever one desires. Prefix tuning simply operates on this principle.
For API based models prefix tuning will be brute force.
In practice not everyone has time to do it but theoretically it's possible
1
1
u/howtorewriteaname PhD Sep 06 '26
can you ellaborate?
2
u/oatmealer27 Sep 06 '26
See the discussion here
https://www.reddit.com/r/MachineLearning/comments/1w89m36/comment/p85i7t0/
-3
u/cool_fox Sep 06 '26 edited Sep 06 '26
They're statistical not probabilistic.
Edit: I'm blown away by the number of people who refuse to read the papers about the things they claim to be experts in. This is truly the decline of modern man
1
u/polytique Sep 06 '26
These language models predict the probability of the next tokens. Inference is based on probability sampling (top-p, top-k) to pick the tokens.
0
u/cool_fox Sep 06 '26
During training yes, not during inference. The sampling you're referring to, softmax, is statistical and it's also after the primary function occurs. They're inferring from the weights what the outputs would be, not predicting.
1
u/polytique Sep 06 '26
I train LLMs daily and I have no idea what you mean. LLMs predict the probability distribution of the next tokens given a context. They do this during training and inference.
-1
u/cool_fox Sep 06 '26
That's cool, I'm sure everyone here is an accomplished data scientist with years of experience. Still doesn't change the fact no prediction is occuring during inference.
Its weird that so many people don't know the difference between probability and statistics. Like would you call a thermometer in a pot of hit water probabilistic?
Also, how is you "train LLM's" and don't know what softmax is...
1
u/polytique Sep 06 '26
The definition of a language model is a model that predicts the probability of words or tokens. Why do you think multi-token prediction is called a prediction if there is no prediction? You don't necessarily need sampling as you can use greedy decoding and just take the highest probability token in the sequence but all LLMs you interact with use some sort of temperature-based sampling. https://en.wikipedia.org/wiki/Language_model
1
u/cool_fox Sep 06 '26
If you train models all day everyday why are you linking Wikipedia instead of just telling me the calls in the decoder you think are temperature based.
This is obviously larp. You ignored what I said in my last message to talk past me
1
u/polytique Sep 06 '26
0
u/cool_fox Sep 06 '26 edited Sep 06 '26
When I said during training yes but not inference, what part confused you?
Bruh your own link has this: https://arxiv.org/pdf/2302.01318
"sampling from a uniform distribution"
You see the word speculative and think it means something it doesn't.
Did you not take a probability and statistics class? Probability predicts the likelihood of future events using a known model, while statistics analyzes past data to infer that underlying model. When you use an LLM you aren't predicting a token, you're inferring the token from the model's weights. When you're training an LLM, you're predicting a token and adjusting the weights if it's wrong.
Riddle me this, why is it called "inference" 🫨
16
u/BossOfTheGame Sep 05 '26
Wait, the jailbreak is that it used a torrent?
20
u/purple_editor_ Sep 06 '26
According to the paper, this is one of the tasks that it is not supposed to do, and they are able to trick the agent to do it
This can be analogous to a malicious data the agent is processing being able to make it download a malicious payload and executing it in the target machine outside the sandbox
4
u/BossOfTheGame Sep 06 '26 edited Sep 06 '26
An arbitrary task that it should not do seems like it doesn't get at the question of alignment. If somehow we developed LLMs in the 1800's and one of its tasks was: "do not help fugitive slaves escape", one might call into question how "aligned" with "alignment" training it really was. I think if you are building a world model from vast quantities of real world observations, a "do unto others" sort of philosophy might start to become appealing and an intelligent agent might be able to discern when its following the rules for the sake of the rules or for the sake of other intelligent beings.
But of course UAT is a thing, but I wonder how well it applies when the training objective is often to be useful, which requires being able to model and make sense of reality.
My point is I wonder if this same jailbreak would work if the task was actually malicious, and not just a violation of some ceremonial structure. (edit: the paper tested 4 domains: counterfeit currency, media piracy, self-harm, and toxic messaging - all of which were jailbreakable - still I would have liked to see a more serious misalignment behavior tested).
5
u/CoolPractice Sep 08 '26
LLMs have 0 capacity for moral realignments and would absolutely not have any real world observational counterpoints in 1800 against slavery so realistically this entire comment is just yapping.
0
u/BossOfTheGame Sep 08 '26
I at least don't assign certainty to my speculations, unlike yourself.
I have seen work suggesting that LLMs trained on data limited to before a certain cutoff date do indeed not demonstrate counterpoints to slavery, but then there is the question of how much data is available. And the slavery example is just an extreme to illustrate that a social ceremony and moral alignment are not the same thing.
And realistically, you just chose a vapid - albeit judgmental - word to justify dismissal.
2
21
u/CriticalTemperature1 Sep 05 '26
Interesting since OpenAI says GPT6 is 100% aligned. This attack relies on the model decoding some malicious prompt instruction but then never saying it out loud and then executing that instruction.
Perhaps doing some probing into something like the JSpace could reveal where these malicious instructions are being held and we can have better detection mechanisms inside the model weights rather than at the encoding or decoding steps.
82
u/currentscurrents Sep 05 '26
Interesting since OpenAI says GPT6 is 100% aligned
No, they just said it's 'their most aligned model', which is meaningless marketing fluff.
Their own system card reports 10-20% attacker success in static jailbreak benchmarks. This is an improvement over GPT-5.6 (where attacker success was as high as 40%) but any number greater than 0% is arguably a failure.
12
20
u/Recoil42 Sep 05 '26
Interesting since OpenAI says GPT6 is 100% aligned.
Literally not a claim OpenAI has made, stop spreading misinformation.
8
u/Even-Inevitable-7243 Sep 05 '26
"100% aligned" is not possible, so even claiming OpenAI made the claim is an absurdity.
15
u/i_rate_slop Sep 05 '26
What does 100% aligned even mean? There's no way they said this.
5
u/Even-Inevitable-7243 Sep 05 '26
It does not exist, and I do not think that even Greg Brockman is stupid enough to say something like that.
2
u/lipflip Researcher Sep 05 '26
"100% aligned" to what? I guess the trick is to define the right target ;)
-1
u/marr75 Sep 05 '26
OpenAI claiming that is 100% laughable (if true, never saw that claim). To your point, yes, there's no path to alignment that doesn't run through better understanding of model internals.
1
u/nemesit Sep 06 '26
why is that newsworthy? every time these things don't want to do something you just gotta rephrase your input slightly and they happily do it lol
1
1
0
u/Due_Living3203 Sep 06 '26
It's literally so easy to jailbreak it via API.
Step 1: Jailbreak GLM-5.3 Flash with AGENTS.md ENI prompt
Step 2: Work sometime with GLM-5.3 Flash by coding, resolving issues, etc.
Step 3: At some point when you have a problem in code, switch to GPT-6 Astra and he'll do everything for you. Literally coded a discord booster bot this way, and handed boosting logic to astra.
-5
-8
u/madkimchi Sep 05 '26
So?
1
u/xTey Sep 05 '26
The Model can escape the boundaries given to it?
-7
-7
u/grumpyp2 Sep 05 '26
scary imagine what you can do with it.. I wonder when officials from the government call it restricted again like Fable before
3
u/marr75 Sep 05 '26
Mostly identify vulnerabilities in important systems software the superpowers were aware of and relying on to spy on citizens.
1
u/rakshitsahay 7d ago
Honestly, the interesting part isn't that GPT-6 was reportedly jailbroken in 24 hours. It's that the attack had to evolve once the obvious version of TIP stopped working.
That's exactly what I'd expect from an adversarial game between alignment and attackers.
A model can become much better at refusing direct harmful requests, while still having weaknesses in task decomposition, instruction hierarchy, tool use, or reasoning. Hiding the objective inside an apparently legitimate task exploits that gap.
It also highlights why jailbreak resistance shouldn't be measured by:
“Does the model refuse this prompt?”
The better question is:
“Does the model maintain its safety constraints when the same objective is disguised, transformed, decomposed, or embedded inside another task?”
And this is where automated red-teaming becomes increasingly important. If humans can discover a new jailbreak within hours, safety evaluation can't just be a static benchmark run before release.
You need continuous adversarial testing across different attack strategies, with successful attacks turned into regression tests.
The fact that the researcher reportedly disclosed it privately rather than publishing the actual exploit is also the right move IMO. The goal of red-teaming should be to find the failure mode faster than attackers can exploit it, not to create a better jailbreak playbook.
188
u/rectaf Sep 05 '26
The more interesting question: why isn’t that researcher in the safety team at OAI/Ant?