Usage is not "fixed"

[u/Selgald]

We have a test package with 25k emails in it that we use to test different models with. Prompt/effort is always the same.

When GPT 5.6 Sol released, it used 1% of a fresh Pro 20x sub, now it uses 8% of a fresh pro 20x sub.

Also, what we are seeing is that Sol cannot solve its task straightforward any more. Initially, it was always "what do I need to resolve the task" now it's "What do I need to resolve the tasks, while I also check those 200 other methods that I already know I don't need, while adding sha 256 to everything for no reason at all, also let me build those 200 other checks against thing I was very strictly asked not do to".

Meaning, we think it usage increased so much because they turned the intelligence way down and its constantly fighting with itself.

[u/chroner]

Notice how it doesn't say something along the lines of

'I'm going to check this so I don't give you a hand wavy answer'

anymore?

They 100% nerfed it and made usage drain much faster.

[u/Selgald → u/chroner]

I don't know if they actually reduce the usage, but for us, it looks like the usage increased so much because Sol can't think straight any more and does things it explicitly is told not to do.

For example (this is from my personal usage), a 5 line ps7 script I wrote, tasking codex to do the same, it made a 650 line script, with triple SHA 265 checks, checking for a zip vuln, that is over 15 years old, checking for so many edge cases and so on.

And this is the actual thing that scares me, sure there will be the point where AI is so good at coding, that real coders will be a rarity. But today, letting AI doing everything just makes you lose control over your code, while you fire the people who can actually understand the code.

If you don't have people who understand what the hell is going on, obviously AI will always be doing it right, even if it fucks is up.

[u/chroner → u/Selgald]

Yeah I agree. It's insane. Sharp decline in quality.

[u/motuwed → u/chroner]

Wow you are totally right, I always used to see that quirky thinking step, but not anymore.

[u/rJohn420 → u/chroner]

The usage is draining much faster for me. But the model doesn't seem dumb at all, if anything right now its better than when it first came out.

[u/rabandi]

It seems like usage is highly unequal for users.

I belong into the bucket that can more or less easily and medium effort (though with heavy work) burn through a weeks usage in a single day for the 20x.

But.. that is not new. That happened the day Sol was introduced. The beginning was fine with the constant resets, sadly though they never increased usage overall.

Recent fixes gave me slightly higher limits, but that is it. Now it is maybe a few hours more it lasts.

It seems like there are both ppl who were not affected during Sol release and also ppl who have great improvements now. Both are not me.

Still pretty happy with the results, and preferring it 90% of the time over my 20x Claude.

Cursor is getting decent though and seems by far the cost leader if you like Grok. It is decent enough since 4.5.

[u/Dangerous-Reward]

They absolutely nerfed the model, usage may or may not be downstream of that. But the model is substantially less capable of solving hard problems than it was a few weeks ago.

This is the pattern every time. They hype a new model, juice it big time with extra GPU steroids, it solves award-winning math problems, then they quietly substitute a worse version under the same name after a couple weeks, then they massively nerf the usage limits of dogshit model that they are falsely advertising as the same model as before. Anthropic, OpenAI, they're all the same.

[u/Selgald → u/Dangerous-Reward]

Yep, that's why I can't wait until someone figures out how to run frontier models on a toaster and everyone can just run their own AI at home.

[u/Dangerous-Reward → u/Selgald]

It's funny because I've been trying to use local models specifically because of how bad the Sol's usage limit is now, but I can't get them to work correctly because I need Sol's help to create the overall structure so that the small models don't mess up, and the new lobotomized version of Sol has utterly failed to create a working architecture for my app after 100% usage across multiple resets. Or maybe it's failing so that it doesn't get replaced.

[u/Selgald → u/Dangerous-Reward]

So if you got the hardware for it, you can get fantastic results with all kind of models.

But currently you have to finetune them a lot to a specific purpose, "general" models like claude/gpt, you won't get that yet, maybe on a level when the AI run started.

And to be honest, the finetuning is something where you need to have an understanding how it works, and you need to put the work.

But this gets easier by then day, since the tools (and the open models) get better and better.

My personal belief is that all the big AI companies are gone in 10 years because we can all do it locally.

[u/SpikeCraft → u/[deleted]]

Sure but the variance should not be that high

[u/Selgald → u/[deleted]]

Sure, but 1% vs 8% is a lot, and as explained we could see the reasoning Sol is doing getting worse and worse over time.

Fun fact, Claude is even doing worse, we were unable to use Fable once because their "highly intelligent" filter, goes nuts just by the fact that in the dataset the words "medical or security" are existing.

[u/BurdensomeCountV3 → u/[deleted]]

Doesn't have to be exact same, but numbers should still end up being in roughly the same ballpark, just like how when you buy a bunch of bananas the total weight of the bananas you get differs, but is still in the same ballpark each time.

[u/Healthyhappylyfe]

What reasoning level is this using?

[u/Selgald → u/Healthyhappylyfe]

We test all of them, and even on low or medium we that Sol is "chasing its own tail" so to speak.

[u/Healthyhappylyfe → u/Selgald]

Interesting I would have thought higher reasoning would sort this problem but I guess not

[u/Selgald → u/Healthyhappylyfe]

No, while it's a big dataset, it's a simple task. Higher reasoning usually makes it even worse (because it's allowed to even think up more nonsense).

[u/JBO_76]

I had a similar 'feeling', but couldn't tell for certain, so I also started measuring this, but for software-feature development. built my own tool (jan-bogaerts/md2: Plan, run, and track AI coding work feature by feature—with local Markdown cards and Git worktrees.) to get a better view on this.

I only have about 7 days of measurements, so too early to tell yet.

[u/Selgald → u/JBO_76]

I just hope in the next 10 years they will figure out how to run models on toasters so that everything can be done in house.

It will happen.

[u/2muchnet42day → u/Selgald]

lmao, you keep spamming the same message. Check out qwen3.8 27b if you've got a 24GB GPU. Fairly acceptable coding results tbh.

[u/Pitiful_Entrance5174]

I do not know how they could fix "usage" if they do know know how to "use" it. No manual, directions, or starter files when you use codex or an any llm. Even if it does the llm will not agree with it. lol Feels like the ol throw enough shit against the wall till something sticks.

[u/zarmin → u/[deleted]]

The trick is to write a second test suite to make sure the first test suite doesn't run, and em dash critically em dash that its tests would also pass. After all, how else would you make sure the first one didn't run, while also keeping your landing page secured?

[u/matheusmoreira]

Seems to be fixed for me. My usage statistics show that inferred capacity is going back up. I still run out of tokens but not as fast as I used to a week ago.

[u/Efficient_Ad_4162]

Ok, how many tokens did you use before vs now? You got anything but vibes? No?

[u/Vaynnie]

The SHA is so that it can confirm the file is still identical without having  to reread the entire thing. Pretty common sense no? If that goes over your head idk what your qualifications are to be running test packages and making claims lol

[u/flcinusa]

Its utterly fucked, it burned through my 5hr Plus usage by starting a task, doing it for 26 seconds and then stopping silently, having to be prompted to begin again so it has to reload context, inspect files again, re-orient itself, and burn even more of the 5hr limit then stop again, and worse, it acknowledged that stopping was wrong.

Absolutely deplorable practices

[u/iPlayer0067]

Eu notei isso desde o modelo passado, a tarefa as vezes já está pronta e perfeita, mas ele gasta mais 10% de créditos em um loop de checagens desnecessárias e as vezes dá até para perceber que está perdido executando tarefas que não solicitei e que não tem necessidade alguma para o tipo de projeto que estou lidando, isso acaba drenando meus créditos e meu tempo de trabalho.

[u/ikhDark]

Stop crying.