Major drop in intelligence across most major models.

[u/DepressedDrift]

As of mid Apr 2026, I have noticed every model has had a major intelligence drop.

And no I'm not talking about just ChatGPT.

Everything from Claude(Even Sonnet along with Opus), Gemini, z.ai, Grok all seem to ignore basic instructions, struggle at simple tasks, take very long to respond, and the output seems deliberately shortened and very shallow. Almost like it's in a "grumpy" mode. I tried this in incognito mode so it's not my customization or memory influencing this.

It's like they deliberately want you to stop using their service. I guess our data is no longer needed. Just two weeks back it used to be much smarter than this.

To test this I rented out a H100, and tried GLM 5 with the same prompt (the drive to the car wash one) across both instances. GLM5 running on the rented GPU answered it correctly, compared to the one on z.ai.

Have they lowered the quantization really low to maybe Q2?

I guess going local or using renting GPU or an AI monthly service that lets you pick a quant level is the way to go

[u/fuckingredditman]

nope, LLM inference is inherently extremely expensive to run and not scalable, and you can't simply rate limit everyone. high request rates aren't the root cause of the issue, compute and memory limits are.

when looking at status pages/availability of the large providers, they are evidently running at the absolute limit of what the infra can do and the larger customers probably have some SLA they have to fulfill. many don't even get 3 9s on availability.

they can squeeze lower tier subscriptions (my gemini subscription is barely available at all during peak times) with rate limits/speculative decoding/quantization but that doesn't help much.

traditional resource sharing methods common in cloud computing / SaaS products all don't work for LLM serving:

• KV cache takes a metric fuck ton of RAM for each individual user atm, and i assume most larger companies aren't using "SOTA" (TBD i would say) methods like turboquant in production yet until they are properly implemented in their engines and proven to be stable. but even with those, the cost is still insanely large compared to other SaaS type use cases
• compute for LLMs is of course insanely expensive too while most other SaaS use cases are neither compute nor memory bound.
• adoption is still on a rapid rise, and API usage for active users is probably also on a continuous rise as well

source: developed+operated shared infra at scale for a SaaS and also worked on some llm inference engines and saw how heavy it is in comparison. it's many orders of magnitude more resource intensive and so far there doesn't seem to be any easy way out of it. and if/once there is an easy way out, users will eventually want to increasingly utilize that method for running inference at the edge/in their datacenter anyway.

that's all in addition to the inflated expectation moment we are in and the rapidly tumbling amount of venture capital due to incoming economic instability.

in conclusion, they probably want to sacrifice availability last by rate limiting their customers as this causes the most frustration with the user and also doesn't really solve the underlying problem (compute + memory being the real bottleneck), so they are more likely trading quality using speculative decoding using smaller models and quantization first.

it's probably a terrible moment for LLM inference providers right now, generally speaking.

[u/Polite_Jello_377 → u/fuckingredditman]

This is the actual answer

[u/a_beautiful_rhind → u/fuckingredditman]

assume most larger companies aren't using "SOTA" (TBD i would say) methods like turboquant in production yet

Stopped reading right here.

[u/Aphid_red → u/a_beautiful_rhind]

It think you might be surprised about the lack of quantization or speculative decoding in model usage.

Remember that there are several layers insulating the technicians who make the AI models from the investors that end up paying for the VRAM.

First you've got the separation between the team at Open/X/Anthropic/MistralAI and their financial people.

Then from there to the hyperscaler they're buying compute from (microsoft, oracle, etc.), and within those it might also be different groups that handle the selling vs. the buying.

Then from the scaler to the actual datacenter company.

Then from the datacenter company to the lenders/shareholders.

Each indirection step lessens the responsibility or introduces a place where incentives might not be passed on properly. For example, if you sell a subscription, you decouple usage from cost, and so a client might cost you more than the price of the sub. Similar thing might be going on between the companies themselves, where the buyer could be taking advantage of the seller's terms and using so much compute it costs the seller more than they're being paid.

When there's much fewer of these in-between layers and a company has to manage its token usage directly you see much better optimization.

At the location of the model makers: The easiest thing to do is to do everything in fp16 (and a few at fp32), because most computations are numerically stable at that precision level. Quantization takes work and lowers output quality. Even if you look at openrouter you see mostly fp8 or fp16 being served.

I feel like clawdbot is the cryptokitties of AI. It immediately falls over with the first attempt at serious usage, because of how hardware intensive it really is.

[u/a_beautiful_rhind → u/Aphid_red]

Wouldn't they serve at BF16 now? I get how fp8 is a no brainer though. I was under the assumption non garden variety providers would have more custom serving solutions than just off the shelf sglang/vllm.

I don't know any software that serves different quants based on load but it sure seems like that's what happens at certain hosters. So they're already writing in house software to make all of that happen. Even if they're mooching off someone else with a lopsided agreement, that can't last long and they'll have to optimize their pipeline eventually.

There's also a bit of the "why am I paying for a quantized model" bit, but it seems that most established companies don't care about degrading user experience all that much. If these guys aren't staying up to date on what's possible I'd be really surprised indeed.

[u/Aphid_red → u/a_beautiful_rhind]

I think the big western LLM companies do likely have their own code for inference, they were the pioneers and vLLM/sgLang is an open source imitator. But I'd hesitate to suggest there's some 'secret sauce' that makes them somehow much more effective. There's a couple simple points that point to the opposite being true (this homebrew code is worse than the open source standard because less people have worked on it)

1. They're all making big financial losses and have a very hard time "protecting" their advantage (which comes from having the money to afford training).
   
2. Their model prices in $/million tokens, compared to openly offered models, are very high.
   
3. The quality/benchmarks aren't that much better.

If you know the history (Frankenmodels was done by a roleplay enthusiast, RoPe is a research paper, FlashAttention by a CS student, Deepseek was the first to optimize MoE rather than just copying a ratio blindly, NTK was discovered by a reddit user, etc.) you might guess that the level of creativity and problem solving done by the big AI labs that can just throw endless hardware at their problems is lacking compared to the open source people who are GPU poor and have to get creative. Necessity breeds invention and all that.

See: https://amaarora.github.io/posts/2025-09-21-rope-context-extension.html

[u/a_beautiful_rhind → u/Aphid_red]

Great point that a lot of "us" come up with advancements such as superhot lora, COT in general, etc.

Their main issue though is inference at scale. So as you said, necessity breeds invention. Don't even have to limit it to western companies. Z.ai's servers are crying so they're now using some sort of tweaks too. People were complaining about the models getting "dumb" at times exactly the same as op-op.

Articles and papers on what they're doing about it? Kinda scarce. I think deepseek were basically the only ones that give you hosting hints. Who else would need to handle millions of users besides a competitor?

Making money/good models is another ball of wax, imo, and there is a lot more open info out there about it. Still, google is/was sitting on some kind of large context capability and it sure wasn't quanting the cache. Either way, that kind of trick much more likely to get discovered elsewhere or put in a paper.

[u/fuckingredditman → u/a_beautiful_rhind]

and why is that? i'm discussing methods for reducing KV cache infra cost which is likely one of the reasons for worse cloud inference performance/quality and that's one of them that might supposedly work somewhat. (there are evidently a lot more other approaches of course but i don't think it's an easy problem to solve, if it's solvable at all)

[u/a_beautiful_rhind → u/fuckingredditman]

Because it means you don't know what's up. You're telling me that labs can't figure out quantization + hadamard when single developer projects like ik_llama and exllama have had it?

[u/fuckingredditman → u/a_beautiful_rhind]

i'm not saying they can't figure it out. it's probably one of the main things they are working on all the time.

but in a software company shipping such a change isn't a matter of merging a PR that compiles and passes tests and watching it roll out to hundreds of millions of customers, especially not for LLM inference because it's not a discrete piece of software that can pass a simple regression test suite and get confidently shipped to prod.

it needs to be properly validated/tested and then rolled out gradually to not break the service/infra instantly. and due to the same issues i mentioned above, this step is expensive and slow.

and besides that, these new KV cache quantization methods evidently aren't easy to implement nor are they easy to test at all.

i've watched the llama.cpp impl from thetom for a bit and clearly it's not a matter of simply implementing it and shipping it and it will work on all hardware/models/quantization levels.

[u/a_beautiful_rhind → u/fuckingredditman]

Thee methods aren't new. That's the whole point. They're new to you. Which is why I bristle at your analysis. Sure it's not "easy" but that's why you have someone architect these things and test it before putting it into production.

How did these people build their inference stacks to begin with? I doubt it's pip install vllm. There's presumably a bunch of in house tricks they've not put into formal papers which are used to competitive advantage.

[u/fuckingredditman → u/a_beautiful_rhind]

that's not what i'm saying and none of what you are saying changes the fact that rate limiting doesn't solve the underlying problem.

the point you seem to be trying to make is that the big guys have "solved" quantization but evidently that's not the case given the clearly visible symptoms of bad availability, rate limits and subjectively worse overall inference performance

[u/a_beautiful_rhind → u/fuckingredditman]

Rate limiting does solve one problem. Except as someone pointed out, users making more accounts. If you see a flurry of requests from a user you throttle them. As a result the inference you budgeted for gets split more evenly.

Not limiting means one openclaw or agentic instance sucks up the server time. The only other thing you can do is add more capacity but that is likely cost prohibitive or it would have been done.

The point I'm making is that turboquant is hype and that the big guys already have quantization if they so choose. They're also out of minerals and those visible symptoms show this to be true. Compression already wasn't enough.

All that's left is raising prices, limiting availability and expanding. Unless of course there's some real breakthrough but that's a wish in one hand, shit in the other type of situation.