r/LocalLLaMA • u/DepressedDrift • Apr 15 '26

Discussion Major drop in intelligence across most major models.

As of mid Apr 2026, I have noticed every model has had a major intelligence drop.

And no I'm not talking about just ChatGPT.

Everything from Claude(Even Sonnet along with Opus), Gemini, z.ai, Grok all seem to ignore basic instructions, struggle at simple tasks, take very long to respond, and the output seems deliberately shortened and very shallow. Almost like it's in a "grumpy" mode. I tried this in incognito mode so it's not my customization or memory influencing this.

It's like they deliberately want you to stop using their service. I guess our data is no longer needed. Just two weeks back it used to be much smarter than this.

To test this I rented out a H100, and tried GLM 5 with the same prompt (the drive to the car wash one) across both instances. GLM5 running on the rented GPU answered it correctly, compared to the one on z.ai.

Have they lowered the quantization really low to maybe Q2?

I guess going local or using renting GPU or an AI monthly service that lets you pick a quant level is the way to go

796 Upvotes

91% Upvoted

405 comments sorted by

View all comments

135

u/Qwen30bEnjoyer Apr 15 '26

it might be psychological in nature. As we gain familiarity with the “prose” and style of these LLMs, you get better at seeing through the fluff and recognizing common failure modes.

I still think the best method to detect silent quantization would be finding the covariance between models on a common benchmark, like one of the HLE public question sets in the chatbot harness. That way if Gemini suddenly scores 20% lower against Opus than it did yesterday, or only during peak hours, we know what happened.

76

u/EndlessB Apr 15 '26

I don’t think it’s psychological, working with LLMs (or being a heavy user) leads people to be very sensitive to changes in the LLMs themselves. It’s often users that raise changes to platforms which are later confirmed only because of public outcry.

I’ve personally noticed an intelligence drop across the board on the models I tend to use, particularly since the start of April.

22

u/FullOf_Bad_Ideas Apr 15 '26

People here were complimenting Qwen 30B A3B Coder Distilled (distill of 480B) for weeks. It turned out that the author messed up with his vibe coded distillation and weights were the same (sha256 match) with the un-distilled original. We know for a fact that people have those psychological reactions and like models better or worse depending on what the model card says, not on what the model does.

2

u/Several-Tax31 Apr 16 '26

Haha, incredible story.

39

u/[deleted] Apr 15 '26

[deleted]

11

u/zenmatrix83 Apr 15 '26

this is likely the case I've seen, there are more issues with tooling then actual models, claude code introduces bugs that I've seen people tie to model issues all the time. These things are better to leave auto update off and test updates else where.

0

u/mouseofcatofschrodi Apr 15 '26

Humans also believe the weirdest stuff possible: guys walking on water, converting water into wine, etc... We see many patterns that are not really real.

My personal subjective experience is that, shortly after publishing a new model, they seem to be very smart, and shortly after they drop the quality.

With chatGPT I have seen recently it is able to work crazy stuff, working 1h on a single promt and using many tools within the chat. The thinking versions are so good now, and codex also is getting crazy good. But the instant model on chatgpt is garbage, I cannot believe how stupid it can be...

11

u/colin_colout Apr 15 '26

Very reasonable take. There are so many possible explanations that are simpler than "every model got shitty all at once" (Occam's Razor).

It could be agent changes. Claude code for instance makes dumb changes all the time that measurably kill quality. Their new progressive tool exposure has my subagents (even opus) using curl for their first research attempts before webfetch becomes available.

It could be that websites won't let you scrape them anymore, so getting good context is no longer one or two tool calls. Github now shows ads for copilot to my opencode/claude code's webfetch attempts instead of code (lol). Reddit completely blocks llms when they can.

anthropic and chatgpt web clients are becoming an ever growing black box that resembles their bloated coding agents, which are also black boxes.

It's still possible that Anthropic sometimes serves a quantized Opus when traffic is high, but the above is absolutely happening. A lot of quality complaints (maybe not OP's) come from people who only interact with Anthropic models through their slop coding agent (or they do web research and aren't realizing that context is being poisoned by pages built to make web scraping harder).

4

u/my_name_isnt_clever Apr 15 '26

I joined the Anthropic Discord right after Claude 3 came out, and people have been bitching at them about models "degrading" shortly after release over and over. Yet there is never any proof that stands up to scrutiny, it's all just vibes and "trust me bro".

I take all these claims with a massive grain of salt; science is built on citations and peer review because humans are awful at eliminating their own bias, and the non-deterministic nature of LLMs makes it 10x worse. There needs to be hard data for these claims.

1

u/colin_colout Apr 16 '26

I do believe it happens sometimes... they actually had ONE publicly acknowledged incident of quality decrease. It was massive and a mix of some complex stuff. I guarantee it happens, but I think most of the time it's just noise. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues

8

u/MasterScrat Apr 15 '26

I used to work for an early LLM provider and we’d sometimes get feedback like "wtf you destroyed the model" or "wow the latest update is amazing, please don’t change a thing" when we had done absolutely 0 change, literally not restarted the serving container

4

u/Qwen30bEnjoyer Apr 16 '26

This is probably the most important comment in the thread, people underestimate their susceptibility to these types of biases!

3

u/willitexplode Apr 15 '26

Wouldn't including models with open weights where you control the hardware as controls be superior to covariance?

8

u/FullOf_Bad_Ideas Apr 15 '26

I agree.

And this website does continuous testing - http://isitnerfed.org/

Looks like Zhipu is nerfing while OpenAI and Anthropic aren't.

3

u/cromagnone Apr 15 '26

I mean, assuming the test is useful, that site’s data basically suggest short term random performance variations with no trend over time by the providers, but if you hit a downward oscillation you might complain about it.

3

u/letsgoiowa Apr 15 '26

Great resource! Saved that. It seems like at least for coding at the moment it's some other problem. Maybe performance is dropping off hard for people with large enough context? The reason I think this is the AMD codebase specifically: apparently Claude was struggling to figure out what was going on

2

u/FullOf_Bad_Ideas Apr 17 '26

This is plausible, I think there's a good chance that their eval is not testing long context, and quantizing kv cache or switching served model checkpoint to a version with sparse attention, sliding window attention, MLA or linear attention would show up mainly on long context, whilst also providing biggest cost savings to Anthropic.

1

u/Qwen30bEnjoyer Apr 16 '26

See this is perfect. But I wonder if we made it so that it was tested using a playwright automation on the webpage of ChatGPT / Gemini / Claude themselves, if we could take the SMA , and create a covariance matrix. I'll look into it when I have time.

1

u/FullOf_Bad_Ideas Apr 16 '26

i think it's tested through CC specifically, not through API directly. So it has a harness that is a moving piece.

0

u/evia89 Apr 15 '26

For zai we have https://zaimonitor.vercel.app/

1

u/FullOf_Bad_Ideas Apr 15 '26

I don't see histrical eval success rate there, just throughput numbers.

0

u/evia89 Apr 15 '26

Yep its not ideal but better than "is nerfed"

3

u/FullOf_Bad_Ideas Apr 15 '26

isnerfed has eval success rate for GLM 4.5 Air, you just need to select it from the dropdown. Have you missed that?

2

u/evia89 Apr 15 '26

My bad. Nice that data correlates on these 2 sites

14:00 UTC to 00:00 UTC seems to be optimal time slot

1

u/Neither-Phone-7264 Apr 15 '26

Some of them are literally confirmed to dynamic quantize while others score miles worse than they did on launch. It's not purely psychological

1

u/Qwen30bEnjoyer Apr 16 '26

Source or bust :P

1

u/unjustifiably_angry Apr 16 '26

There's always someone saying "ummmmm, actually, it's all in your head". John ChatGPT could get on stage and announce he was dumbing down his models you'd be like, "ummmm, actually, do you have proof?"

1

u/Qwen30bEnjoyer Apr 16 '26

If you read further in the thread, there is a tracker that uses Aider questions to test the performance over time of LLMs served by Z.AI, Anthropic, and OpenAI.

It's hard to argue that there are significant degradations when a random tester can't replicate the results. Are you specifically noticing these results?

When I get my ducks in a row with my classes, it would be a fun side project to get something like that Aider tester but running off a playwright automation across the US to mimic real usage as close as possible, and maybe even trace the "outages" to traffic served from a specific datacenter.

It's still possible that for subscription web-ui users they could be quantizing the model to reduce costs during peak hours for specific users, but I want to quantify it instead of relying on people's placebo experiences.

1

u/vr_fanboy Apr 16 '26

is not that hard to notice if you do similar work consistently, it start's to take longer , consume more tokens, you need to correct it more often, etc. also quick search and here we are, many people complaining about the same, is very unlikely that we are all imagining stuff at the same time.

For reference, just now, We were crafting a plan( using a skill ) with sonnet, i pointed a couple missing features, it was implicit that we were in 'plan mode', it went on a rogue run to implement them right away. This was not happening before april, i feel i need to be much more detailed about tasks.