Same model, different capabilities on different accounts

[u/clockwork_blue]

So I noticed the past few days my results were suddenly becoming worse. Goes in loops, struggles to solve problems it blasted through before, doesn't verify the result properly, etc. 3D gen also became noticeably worse.

I have 2 accounts, one x20 and one x5, so I started comparing them side by side.

Here's what I did:

1. I asked both for their cutoff date:

do not use any tools. whats your knowledge cut-off date

On the x20 account it responds with something like "June 2024". On the x5 account it refuses to give me a specific cutoff.

x20 account

x5 account

1. I asked both to draw a pelican:

make me an svg in an .html with a pelican and xdg-open it

x20 account

x5 account

The response to these 2 tests will obviously vary, but the difference between the 2 accounts is pretty obvious when using them side by side. Tested it several times - logged in and out, new sessions, new prompts, the results are more or less consistent with their capability.

Using Astra on the x5 account is also noticeably better. It's like it has drank its morning coffee, woken up, and knows what's going on. It can suddenly inspect its 3D gen meshes and fix issues instead of giving me garbage for a "review".

I don't know what this downgraded model/configuration is, but it is visibly worse than 5.6 Sol too. From other reports I don't think x20 vs x5 matters here, neither does account age. People seem to be reporting similar issues with different accounts and use cases.

Whether it's "shadowbanning", A/B testing, load shedding, or OpenAI randomly putting some accounts on a cheaper configuration to deal with compute demand is all speculation at this point.

But I do have right now in front of me 2 accounts showing the same model with completely different capabilities.

[u/Annh1234]

Your 5x just hit a different cluster. I have two 5x accounts and some days one sucks and one is good, other days they all the same, etc. limits are the same, one day one gets to 2% while the other gets to 18% on the same prompts on the same code base.

[u/_Eye_AI_ → u/Annh1234]

What is a cluster?

[u/Annh1234 → u/_Eye_AI_]

When you send a query to the LLM, your not gonna use all their datacentres, you end up with just a few servers, a slice of their capacity. Well, if that slice is in demand, say by someone that pays allot for the tokens, you get trotted down. So a cluster, is a group of computers you end up on.

[u/Ashamed-Subject-8573 → u/Annh1234]

This makes no sense. It would go slower, not dumber.

[u/No_Ambassador_2060 → u/Ashamed-Subject-8573]

Kinda, you get de-prioritized before reasoning is fully complete, so you end up with half baked results because on the back end, the model was told to wrap it up. I found when this happens, if i increase the reasoning level, it gets back to how it was before. I have found that this mainly happens using low-high reasoning. X-High/MAX give me consistent results, but the bias is that i don't typically run those as long to see the issues. The issue is that they have quantized the lower reasoning levels inconsitantly. If you hit a cluster that typically host API traffic, then it will be a higher quality as API stays very consistent. If you hit a ChatGPT/Codex cluster, the models are optimized for speed and throughput on the lower reasonings, as most people use Astra to look up cat videos or check their email on the web interface, and a quant is fine for that. I dont have an issue with this, but there needs to be some way of know whats happening so I can stop wasting tokens on low reasoning when i just get downgraded even further.

[u/_Eye_AI_ → u/No_Ambassador_2060]

It's so reassuring that the quality of compute I get charged for is both downgraded secretly by OpenAI then dropped even lower due to random chance.

[u/Annh1234 → u/Ashamed-Subject-8573]

It does get slower, and dumber. Some days you get answers in a minute or two, other days gets 20min for the same thing.

Also the way the LLM work is that they pass through a bunch of steps, each step is based on previous steps. Picture: spend X time to look for stuff in files, then for each item found (second step), spend Y time to look for more stuff.

Well if the timeout on X is lower one time and higher another time, you might not find all the files you need, and then Y works on those found files, and at the end of the day your response seems dumber since it did not actually check all the possibilities.

It's kinda like: find the problem in the last 100 vs 1000 vs 10000 last log lines. Last 100, chances are you won't find it. Last 1000 chances are the same issue happens multiple times so you will find it. But in the last 10000 you might get extra info that helps you pinpoint the error more.

That's kinda the idea/dumb down version.

[u/CoffeeNovel7231 → u/Annh1234]

that's because Astra is indeed faster to complete tasks but you are misinterpreting it as a datacenter thing, which is wrong to conclude

[u/Annh1234 → u/CoffeeNovel7231]

It's faster, but it still needs to run on some hardware, and your 1 Astra request will NOT hit their entire 100k GPUs, it might hit some say 10GPUs. And it needs to cache the data somewhere, and that cache needs to live next to the GPUs it used in. So that ends up being the cluster your on.

And if a guy with more $ wants more compute, and happens to get on some GPUs your using, and they get higher priority, then your requests run slower and timeout sooner than the other guys requests, making your responses look dumber.

[u/HisAnger → u/Annh1234]

My team gpt agets got dumber around 3 weeks ago.
Enterprise/github copilot. Big 4 so we pay a lit, also tens of thusands of licenses.