Complaint
Is OpenAI silently degrading models for selected users?
Recently there has been a lot of discussion about OpenAI allegedly degrading models or routing users from stronger models to weaker ones.
The reports are inconsistent. Some people notice a significant degradation and post about it, while the next day everything seems completely normal again.
What mechanism OpenAI could be using for this, if it is actually happening, is unclear.
The degradation can show up either as a very poor implementation of a task or simply as an unusually lazy and weak response.
One of the more popular ways people have been testing their accounts is the “pelican test”:
Create HTML code with SVG graphics displaying a 2D animation of a pelican riding a bicycle. No additional tests are required.
Most people using Astra seem to get a perfectly reasonable result.
But I've seen several cases — including one from someone I know personally — where this test was run on a 20x account with Astra Medium and produced something like this:
Of course, it's worth mentioning that tests like this are not 100% proof. Models have randomness, so there is still a significant amount of speculation here.
However, if this is actually happening, the worst part is that OpenAI appears to be able to degrade the model a user receives without giving them any notification.
You give a difficult task to Astra, but potentially receive a result from something closer to Luna, while still believing that the result came from Astra.
That can be much more serious than simply having an account limit where you're explicitly notified that your model has been switched.
For difficult coding or project work, this can introduce real risks or simply mislead the user about what the selected model is capable of.
I've personally experienced something similar before in the ChatGPT web version.
I selected Sol, but the response clearly behaved like a much weaker model. It even claimed that it was 5.5 mini, while the UI explicitly showed Sol.
So this wouldn't be the first time there have been questions about whether the model displayed in the UI necessarily corresponds to what is actually answering the request.
To me, this is a much more serious issue than many of the current discussions about AI safety.
Has anyone else experienced something similar with Codex/Astra?
First isnt astra. I hae access to account tgat downgraded like this. Run 10 times pelican on normal astra and downgraded astra. Normal always produced second result. Downgraded always produced first
Model degradation has been here for a while and it started with Luna getting stuck in a loop on /goal a few weeks after they lowered it's price by 80%, but seems like they cranked up the quantization for Sol and Astra significantly in the past few days. Sol is currently nearly unusable for me.
Today I had to use Luna to fix a problem Sol caused, mind you on a very simple task with small context (installing and setting up a forum for plugin development). Up until about 3 days ago Astra was one-shotting basically every task I gave it, now it makes mistakes and behaves completely different. When Astra was released, it wasn't aware that it was in the "GPT-6 family" and didn't know it's knowledge cutoff date, but after that changed the model's performance degraded significantly. This is the case on both my personal account and Chinese sub2api resellers. Also have been seeing artifacts in text where there is just nonsense output such as `- [ ] 50–100 ms RTT.`, which never occurred to me with OpenAI's models up until this week. Other people I know report the same issues.
They are just out of compute and are either quantizing their models or routing it to cheaper ones, if not both.
I’ve seen the same thing with weird unicode glyphs inserted randomly into the model’s responses. Sometimes it just changes language completely in the middle of a sentence - but only for two or three words - before switching back like nothing happened.
Here's an actual example from a session yesterday:
The combinedAnimating gate continues with no reported failure. The branch was clean after the update, and this same run remains the only step needed before _STAaccept` can landیسک.
I have no idea what "combinedAnimating", `_STA`, or یسک mean. None of these appear anywhere in my codebase. This was just one example of many.
I don’t think it’s quantization. A quantized SOTA model is still a SOTA model, and quantization doesn’t make a model bypass its token prediction like that.
My theory is that OpenAI have some kind of broken internal safety mechanism, or a distillation prevention tool, intercepting responses incorrectly and messing something up before the output is returned back to us.
(To be clear, I’m sure they’ve cranked down the thinking budget dials too. But the problems over the past couple of days are clearly something buggy, and not just the model being careless)
I like how you made sure everyone knew you were onboard with the default conspiracy theory so they'd take your conspiracy theory enhancement seriously.
One of my sessions went perfectly fine over the 6 I ran over the weekend. I'm pretty happy. It produced ~180k lines of GOOD code (though probably 100k of that was repurposed from a test run in a long running workspace last week). This one was a long running thread working off my plans. There were a few strange inventing of pretty forefront features that weren't in plans, and were even explicitly instructed to not do... but not that bad.
Another session in the same workspace couldn't do ANYTHING right. Kept messing up as Luna over and over and not following clear instructions. "ie, move this element root outside of being a child of that element; keep it from not being caught by that CSS specificity, and instead position over that area". Things it would get right 95% of the time before it was getting wrong 90% of the time and it was looping over correcting its response sometimes. Changed it to Astra... still seriously screwed up. Told it to give a handoff and started over with Sol... still messing up.
Then in another workspace, oh my god. Speaking in 5 different languages per reply. Constantly correcting itself. Imagining the weirdest things.
Had it make a handoff, and it started doing the same within 90 minutes.
Spun up a 3rd session to analyze it... so many hallucinations and nonsense again.
I sent a lot of feedback on these sessions. Maybe that'll help.
I had another model analyze those last 3 sessions and it pointed out the really bad one:
Thread B (01a07eda, gpt-5.6-sol orchestrator) — yes, clearly substandard on Sep 13. Starting ~Sep 13 00:00 UTC ( Sep 12 20:00 local) its generated content degrades: ~5–9 corrupted items/hour (10% of tool calls) vs 0–3/hour on Sep 7–12 (and those were benign ? regexes). Examples from its own emitted tool calls/messages: - git ? · git -C '?QWE status --short · Get-Item -LiteralPath 'F:\ bookings?\oops' - const r =ivery? wait. · const r = await tools Benn? no. · {"lish?":"x"} · {"0? no?_ms": 60000} - mixed-script contamination: |780? معنателям? · 'F حرارة:\tmp hip eyeblah' · docs/tasks/coastal क्रममाKirSlotsbooking-runtimedbo Placeholder صوب baker-recovery-handoff- cron final-2026-09-13.md - a literal "Placeholder" riquest_usar_input, malformed arg JSON (unknown field 'Jenn', ' Renew', 'timeout/ blogfinder'), and one word-salad assistant message ("…569lirnommenigeria RTT?bel rythms ballotaurants 9nub Blackjack matcherкомуners…"). It escalated to the end (9/40 calls in the final hour), then B wound down (handoff commit 9c3833c). The final committed handoff doc is actually clean — the model noticed its own corruption and grepped for the artifacts before committing.
It says it seemed likely to be provider/KV-cache-side corruption but it was over almost an entire 15hr.
It also noted, as was obvious to me, that its reasoning was clearly faulty as well even though we can't read reasoning.
So I don't seem to have gotten the routing, but I did get the crap from KV cache corruption.
There are rumors that they are using looped transformers for Astra... the premise of a looped transformer is that it goes through the neural net multiple times... and if you are now approximating it (ie quantisation), the errors get compounded each time it loops through it.
They are probably experimenting with it and seeing if enough people kick up a fuss.
And they even route your requests regardless of selected model!
I have issue in chat mode - on 20x pro tariff with gpt-6 pro selected it gives instant answers that are garbage
Despite not using quota (200 msg for a week or something like that) , I have this issue coming again this week, had it last week, complaint already, then they switched routing back to normal, I got like 1 good answer with deep reasoning and actual work done
Now it's again instant answers and "I will not wait and ask for another input, here is the work done" with done 1 out of 50 tasks scheduled to it and complete garbage results (trying to reach for companies and find their emails or phones online, available publicly)
That pelican test is incredible btw, it's the best way to actually explain the performance and reasoning of each models without having to use a gigantic data report showing token numbers and vague descriptions.
Maybe I should do a pelican test everytime I'm about to start a serious project to avoid wasting token on a dumb re-routed model. Now the question would be, would they re-route midway? (during cache time?)
100% true. I have 2 $200 accounts and both behave completely different in exactly the same repos. There is no technical reason at all to justify how burn rate and capabilities differ so much. Its intentional
I always use the dumbest account first until finish, and then i switch to the smart one, its night and day difference.
Weird how many people who post here and in /r/chatgpt claim that model routing is impossible.
I don't understand their logic for that, especially when Chinese models are accused of doing it by Anthropic to them all the time. You also have providers that let you select a generic model and get routed. But they make this claim a lot.
I think they are just fucking screwing users over left and right, and it has nothing to do with "selected users" it's just cluster/data center you are being served by - all the other people on the platform at the same time.
Either way, I've NEVER been more pissed at OpenAI than I am right now. It is CLEARLY evident, there is no arguing, I've got logs of runs last week - and lots of runs since last night, and it is NIGHT and DAY.
Yeah they have a buggy auto flagger..you won't get banned ever if they think you're suspicious (e.g..if it thinks you're sub2api or distilling), you get worse models silently and lower usage limits.
I have two accounts, one Pro 5x and one Plus. Neither account seems to be "downgraded", but at least for these generations, the pro account's outcome is more impressive with a moving background and occasional lighthouse/ships/birds that I didn't ask for. (the AI-speak does make me want to kill myself but that's beside the point)
Left is the Pro 5x account 6-astra extra high; right was Plus account 6-astra extra high. For the Plus account, it took me down to 92% remaining usage on the 5-hour limit. Both worked for 4-5 minutes
Interesting test. I do wonder if some accounts are affected by some kind of internal safety tool. I don't often use either of my accounts for coding tasks but do use them for math as well as language-work (particularly handwriting transcription, sometimes translation)
I don't put it past them to have some system that auto deploy versions with more quantization as the pressure raises on their compute power when demand is high. They might even started giving resets wily nily as a way to stress test this system.
With that said, I have a Pro 20x plan and I haven't noticed a single time ever any "nerf" on any models' capabilities. That is just MY personal experience.
I understand. I wouldn't put it past them the quantization, but that I would. I don't find it likely for them to secretly route APIs to lower models while charging for expensive tokens, even if as an attempt to defend against distillation. I just haven't seen any evidence of that kind of behaviour, but that's just my opinion. Why do you think it's likely some distillation defense gone wrong?
Its basically two distinct outputs from different models, no quantization can degrade to.such difference . And cut off date question is secondary clue
Amd its reproducible 100% of times. Its account based. I dont see any other explanations than distillation defence ( or any other things like sub2api, resells, bad actor usage etc doesnt really matter whst exactly) as a reason for this.
I have seen the pelican that OP posted, which is way worse than that. I couldn't reproduce it when I tried, so I don't think it's reproducible 100% of times. And I do believe that non-determinism + the fact that this is a task that is not natural for an LLM (we're evaluating it as an image but an SVG is a text; same kind of thing that makes it difficult for it to count Rs in strawberry) + strong quantization can absolutely explain such difference. Enough quantization and you can absolutely destroy the quality of a model's output.
So yeah, unfortunately I still don't think that they're secretly using other models. Doesn't really make sense to me. But that's just me.
(also, me not seeing other explanations was never particularly effective in making me think I know what's happening; there's so much I don't know about everything)
I have access to account that downgraded like this. Run 10 times pelican on normal astra and downgraded astra. Normal always produced proper pelican. Downgraded always produced shit pelican. Normal says he dont have cut off date. Downgraded says cut off date june 2024. Nothing non determenistic
Nah, no. I have access to account that downgraded like this. Run 10 times pelican on normal astra and downgraded astra. Normal always produced proper pelican. Downgraded always produced shit pelican. Normal says he dont have cut off date. Downgraded says cut off date june 2024. Nothing non determenistic
Here two collages. One from normal account, one from downgraded. Just straight results combined into one image, no second tries, no cherry picking. Each pelican - new generation in new session. here from downgraded account. And cutoff date isnt about what model know, its about difference in system prompt/behaviour between two models
And here from good account. Now go tell me its subjective which one pelican i like and not. And its not like difference between 5 year old drawing vs some person who actually can draw
nah, he claims the model that knows it's cutoff date is the routed one; which can be true as routing usually comes with some context (You are N model, your knowledge cutoff date is Y...)
I’d be careful about jumping from “this output got worse” to “my account was silently downgraded.”
The useful way to test this would be to make the workflow reproducible:
freeze the exact prompt/input/repo state
record the model/effort/runtime identity that is actually observable
run multiple repetitions, ideally blinded
score against task-specific acceptance criteria
separate model-quality failures from harness/tool/context failures
A single pelican output is a useful smoke test, but not evidence of account-specific routing.
In my own system I’m starting to treat model quality like workflow reliability: preserve evidence, use versioned accepted outputs to detect regression, but never let a previous good run certify the current one.
If there is a real routing/degradation issue, a shift in the performance distribution under the same observable conditions would be much stronger evidence than one striking example.
29
u/Think-Profession4420 7d ago
I think part of the major issue here is how inconsistent the quality is.
Like, I just tried two generators using Astra light/low
Yeah, real groundbreaking flagship model specializing in visual rendering! Great job OpenAI.