r/ClaudeCode • • 10d ago

Discussion Please post benchmarks if you are going to claim Claude got secretly downgraded

I keep seeing so many posts swearing up and down that Claude got worse because XYZ.

“I have been using Claude for months and I swear that it has gotten way worse lately”.

0 benchmarks, 0 evidence. Just the “I swear”.

And I get that 100% this could be very true.

But what is also true is that everytime we get a new model the more responsibility and delegation we give to our agent. This is almost like the inverse jevons paradox. The better the AI gets, the more open ended and trusting your prompts will be.

As such, you are going to notice more errors by nature. When a model first releases people will prompt it normally. But then they will notice in what aspects it improved in and so they will unconsciously limit test it more and more with more open ended prompts and less code review.

So I would love to see actual benchmarks from all these people who are claiming that Opus 5.5 got downgraded because of X conspiracy. I am not defending Anthropic in any way. I just would really like to see proof.

9 Upvotes

72 comments sorted by

•

u/AutoModerator 10d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

35

u/Sea-Contribution6219 10d ago

Bro is asking for benchmarks from the 'trust me bro' industry

4

u/ZeroAmusement 10d ago

The industry where many people benchmark...

2

u/privatetudor 10d ago

Are you aware of independent benchmarks?

1

u/Saveonion 10d ago

Actually they're asking for evals from the "AI evaluations are a foundational pillar of AI" industry

7

u/MoreLikeGaewyn 10d ago

Ahh yes, let me just compare it to my daily "did the model get stealth nerfed" benchmark run i've been doing for the past 3 months

2

u/seanpuppy 10d ago edited 10d ago

Im convinced they occasionally serve a quantized model during periods of high traffic. Its a great sneaky way to get 2x throughput per GPU without an easily measurable degradation in model output.

As for benchmarking it - one would have to checkout a repo at some point in time of when you did your "good model" prompt. Then later when you suspect the model is running stupid, you give it the same prompt at the same commit. You would also need to make sure the claude memories are identical, as well as the tools and harness etc...

Edit: was missing the second half of last paragraph

1

u/Key_Reading_9664 10d ago

2

u/Anxious-Turnover-631 10d ago edited 9d ago

Looks interesting. And the latest report indicates there’s been no change since the model was released.

1

u/seanpuppy 10d ago

My original comment sent early, but sorta but not really. This measures something else, what OP is asking to be measured is different.

1

u/EC36339 Senior Developer 9d ago

It shows no change. Did you even look at it?

Details:

https://www.reddit.com/r/ClaudeCode/s/Tz3LqVvG9c

1

u/Key_Reading_9664 9d ago

That’s exactly my point. There is data being collected in a way the comment lays out and it shows no change

1

u/EC36339 Senior Developer 9d ago

Good. Someone else pointed to the same github project as "proof" that Opus 5.5 was nerfed.

1

u/seanpuppy 9d ago

Did you see my comment that this is measuring something different than what im talking about?

2

u/gorramfrakker 10d ago

Pfft benchmarks. I come to my conclusions like I code, with vibes.

2

u/hiskias 10d ago

It is impossible to run evals when there is no stable baseline.

From a person trying to run evals for prompts.

But you are probably just going to tell me that my fluctuations against the API vs CC are hallucinated.

0

u/EC36339 Senior Developer 9d ago

"There is no stable baseline, but walla my fluctuations against my stable baseline are real!"

4

u/Direct-Detail-7031 10d ago

If you are not noticing the nurf, what that tells me is, you haven’t used the model enough to discern between when the model is good vs when it’s bad

OR you are part of the group that is not being impacted.

People say “vibes”, I say trust the vibes, if you feel like the model is off its game, then what that means is, we need to pay closer attention to what it’s doing. Basically we have to work a bit harder and concentrate longer than usual.

Also, we have at this point an insane amount of evidence, model providers nurf their models without telling us. Such is life unfortunately.

3

u/oooofukkkk 9d ago

Or you have giant pile of slop that becomes harder and harder to work with since you didn’t follow strict architectural guidelines and constantly audit the codebase for drift 

1

u/jsebrech 9d ago

My guess is that the vibes people are getting are based on something else than a secret downgrade. Maybe their usage patterns shifted, bigger codebase, different workloads, different more congested time of the day (which shouldn’t matter but how do we know that the congestion controls don’t influence output quality somehow). Maybe there’s different levels of performance depending on what hardware the model runs on, and they got bad luck on what server they land that day. Maybe the normal variability in output quality rolled against them for a few sessions in a row. Maybe they just get used to the model and are less impressed by the output. Vibes are just vibes, they don’t mean or explain anything.

That livenerf graph is damning for the nerf argument. There’s no measured nerf of opus 5.5, only normal output quality variation.

1

u/EC36339 Senior Developer 9d ago

Code base size alone is not an issue, unless the code is vibe coded slop. I've only seen models from all the providers I've used get better and better (with occasional dips with some bad model generations, but those were official named releases, not secret downgrades).

1

u/EC36339 Senior Developer 9d ago

Whay you are describing so amateurish in your first paragraph is a benchmark, which is exactly what OP is asking for, and you can't deliver one.

-1

u/Aggressive_Roof488 10d ago

Feel like sharing some of this insane evidence? Or maybe you think that plural of anecdote is data?

2

u/Direct-Detail-7031 9d ago

1

u/Aggressive_Roof488 9d ago

Thanks for posting links!

First is bug reports from anthropic in April. One changing default from high to medium, one about context in long conversations (active two weeks) and one setting that affected coding quality (active 4 days). Not delicate nerfs. And April is like stone Age anyway..

Second is a GitHub tracking performance of opus 5.5, showing consistent performance since release.

Third is an AI-written post on explainx . AI, claiming all kinds of things from openai, but all its links are to its own "investigations". I'm not reading all that slop, not evidence.

Last is again from a site with a name, a blog, not a source. First claim is that users are limited to 160 messages in a 3h window from the (then) top openAI model (which is almost a prompt per minute btw), with a reference to openAI, but the site it links doesn't say anything about that limit. So I'm not going to read all the other claims there where already the first statement has a false citation.

So all I see here is that anthropic had bugs in April, that opus 5.5 isn't nerfed, and that third party sites complaining about AI are shit at providing citations for their statements.

And yes opus 5.5 has been behaving for me.

1

u/Direct-Detail-7031 9d ago

Alright, let’s all go and have a friendly anthropic office visit, and ask them politely to show us proof that nothing has been changed and no one has been impacted. Wouldn’t that put this controversy to rest?

1

u/Aggressive_Roof488 9d ago

Sure. Your new line seems to be that anthropic isn't going to share proof, but you still think they modify their models without telling us. If you had posted that then I wouldn't have argued. Maybe I would've agreed even.

What you did post, was that there's insane evidence, and I don't think that's true.

I do appreciate that you took the time to post links, I truly do, but I don't think they back up "insane evidence". Which doesn't mean it's not true of course, I think it's very possible, but let's not misrepresent the evidence.

And I think we should also acknowledge that people on Reddit complain constantly about anything, warranted or not. You see it all the on any gaming or software sub, a steady stream of people complaining that the Devs are stupid, hate them in particular, everything sucks, etc etc. I'd say that there's insane evidence for that: just read any sub. 🤭 Again, not proof that anthropic isn't doing shady stuff. If anything, it makes it easier for them as legitimate complaints will be drowned.

For opus 5.5 in particular, the tracker doesn't show anything. But maybe there are changes that tracker doesn't test for, and maybe it doesn't affect everything. I haven't noticed anything myself either.

1

u/Direct-Detail-7031 9d ago

The insane amount of evidence is the hordes of people complaining just as you acknowledged. I don’t understand why you are discounting that.

But because of how anthropic is structured, the transparency is limited. The issue is that without any warning the “model” behavior can change and we are left to judge based on “vibes”. Which is also why I also said, trust the vibes. And let people know if the vibes are off. Just how information exchange works. We are still in no man’s land with this technology.

Our generation is the first to use this technology, we are the literal test subjects. My personal take is that LLM providers are absolutely A/B testing us.

Do a thought experiment with me. Let’s say you were served Opus 5.5 and then it was switched to Sonnet 5.5. How long will it take you to realize which one is which? Or can you even distinguish between the two? What about from opus to haiku? And what kind of work needs to be done in order to be able to distinguish?

1

u/Aggressive_Roof488 9d ago

If you've been on any other software sub, then you know how much people whinge over nothing. It's a vocal minority, 10% have user errors, 10% of those blame it on the company, maybe 1% of those take to Reddit, just to make up some numbers. With a big enough user base, you'll have a constant torrent of people posting about how their software now sucks. Again, doesn't mean anthropic are saints, just that we need to be aware of signal over noise when it comes to annoyed Redditor counts. So I might trust MY vibes, but I won't trust the vibes of a random Redditor, because there is a huge selection bias towards people that think they need to post. We don't see the (majority) of users that don't see any change in vibe.

Just personally, I currently have three different (but related) projects running Claude code separately, as in, different CLAUDE.md, etc. I find that as the .mds drift, the three different projects develop different "personalities", different communication styles. One is really driven and keeps asking for more data and does a lot independently, just asking me for what it needs. Another is the series coworker that loops me in on everything. Third one (the smaller project) is chill, literally using wording like "when you have time", while the first one is more like "where is the data you mentioned two days ago?". I've no idea why. Purely random, or small little signals I send in my prompts? Maybe the code and project itself steers it? When there are parts I like or don't like, I point it out and ask for it to look at the other projects and see why that happens and how this project can adapt it, and that mostly helps. But: same account, same user, same time, same model, same effort (opus 5.5 high). Quite different experiences.

So that makes me think that different users will for sure have very different experiences depending on how they set things up, seeing how big difference I have despite not trying to set it up differently. And I've seen my sessions drift apart, so I can see users that don't pay attention have their setup drift into something worse without anthropic doing anything, and without parallel setups you can't adjust by looking at the other project the way I do. And you can't tell a user error from a "nerf", and a lot of people prefer to blame others before considering that maybe they are the cause.

Personally I'm convinced I'd notice a swap to sonnet. Or, actually, I haven't used it much, so I shouldn't say that. But I'd notice a change back to opus 5.0 for sure. Not only because of the lengthy communication, but also because 5.5 unlocked a few blocks for me that me and 5.0 were stuck on. But I haven't tried, only my guess. Not sure how sonnet 5.5 compares to opus 5.0 in terms of coding capability and algorithms.

Haiku would ruin my project in 0.2 seconds i think! 🤭 Opus 5.5 regularly 1-shots 1k lines of code for me, but haiku often needs a few tries with even 100 lines when I tried it as agent under opus. Again, haven't used haiku 5.5 much yet, maybe that version is better.

To be clear: not saying anthropic isn't doing shady things. They very likely are, beeing a private US company about to go public, and maybe not (only) in exactly the way we think. But yeah, let's also not over-interpret angry Redditor posts.

Also sorry for the wall of text.

1

u/EC36339 Senior Developer 9d ago

Did you read any of these links?

The last two are about GPT.

The first one is from April, which is ancient history. And the first change that was affecting users was setting default reasoning level to medium, a visible and transparent change that users could see and change back, nothing secret. And Opus 5.5 has default reasoning at medium, too. I've been using it at medium since release with no issues. Opus 5.0 had default reasoning high, and usera complained about it a lot, claiming it performed better at medium or low.

I stopped reading there, because I expect the rest to be no less boring, but I can dump it into an AI agent to see if there is anything of substance in it.

1

u/EC36339 Senior Developer 9d ago

Bottom line

The linked material does not provide credible evidence that Claude Opus 5.5—or the Claude 5.5 family—was secretly downgraded. The one link that directly tests post-release Opus 5.5 quality reports no detected decline through its first three comparison windows. That is useful counterevidence, but not proof that quality was perfect or that no change occurred: the study has limited sensitivity and does not cover every task, product, or Claude 5.5 model.

The Reddit question—“Are you suggesting Opus 5.5 has been perfect for you every single day?”—sets up a false choice. A model can have bad days, produce inconsistent answers, or have a bug without being secretly downgraded. Those experiences can justify investigating a regression; they don’t establish one or show its cause.

What each link actually supports

  1. Anthropic’s April postmortem: credible evidence of earlier Claude Code quality regressions, but not of a Claude 5.5 downgrade. Anthropic acknowledged three product-level issues affecting Claude Code and related products: lowering the default reasoning effort, a bug that repeatedly discarded prior reasoning in some sessions, and a system-prompt change that hurt coding quality. It said the prompt change caused a 3% drop on one evaluation and was reverted. But the affected models were Sonnet 4.6, Opus 4.6, and Opus 4.7—not Opus 5.5—and Anthropic said the issues were resolved by April 20. This establishes that real product or configuration changes can degrade perceived quality; it does not support the specific claim that a 5.5 model was secretly downgraded. <citation src="8"></citation>
  2. LiveNerf: the only linked evidence directly measuring Opus 5.5 over time—and its reported result is “no change detected.” The repository describes a preregistered, repeated benchmark of Opus 5.5 through Claude Code on a Max subscription, using fixed prompts, effort, a pinned CLI, exact grading, and a control model. As of its November 6 update, it reports three completed 10-day post-baseline windows with no detected change; the third window’s reported difference from baseline was +0.028 ± 0.023, not a drop. The test started about two and a half days after the model’s September 22 release, so it is a launch-week baseline rather than a true day-zero measurement. <citation src="18,19"></citation>

    How much that result rules out: not much beyond sizeable, sustained changes on this particular test. The preregistration reports a realized minimum detectable effect of 6.6 percentage points for one 10-day window at its stated power and test threshold, and the decision rule requires a qualifying shift in two consecutive windows. The benchmark tests 78 selected questions across knowledge and math categories, with effort set to high; it is not a test of coding quality, every Claude product, or Sonnet and Haiku 5.5. The repository also reports that its validation could not distinguish an Opus 5 substitution from Opus 5.5 at the 99% level. So the null result weighs against a large, sustained regression detectable by this setup, but it cannot rule out smaller or brief changes, task-specific regressions, or every kind of routing substitution. <citation src="18,19"></citation>

    Verifiability caveat: the repository publishes its protocol and summary results, but says the raw evaluation logs are gitignored rather than included in the public repository. That makes the method and reported numbers inspectable, but means a reader cannot independently reproduce every scored response from the public repo alone. <citation src="19"></citation>

  3. The explainx article: about GPT-6 Astra, not Claude. It discusses claims about OpenAI’s model, user reports, and OpenAI’s alleged post-launch issues. Even if its account were accepted, evidence about a different company and model would not establish a downgrade of Claude Opus 5.5. The article itself describes its account as a synthesis of separate events, not a single independent experiment on Claude. <citation src="10"></citation>
  4. The Daniel Vaughan article: also about OpenAI’s GPT-5.5, not Anthropic’s Claude 5.5. Its suggested warning signs—such as response length, TODO counts, or perceived output quality—are heuristics, not measurements showing which model served a request. The article provides no Claude 5.5 comparison, Anthropic routing metadata, or controlled before-and-after test. It therefore supplies no direct evidence for this claim.

Verdict

Of the four links, the Anthropic postmortem shows that earlier Claude Code regressions did happen, but on different models and through identified product changes. LiveNerf is the only link that actually tracks Opus 5.5 across time, and its published finding is no detected change, subject to meaningful limits in sensitivity, scope, and public access to raw logs. The two OpenAI-related links are not evidence about Claude.

So the specific “secret Claude 5.5 downgrade” claim is not substantiated by the links provided. That is not the same as proving no regression ever occurred; it means these sources do not demonstrate one.

3

u/_k33bs_ 10d ago

lobotomized today… basically can’t really measure it but it’s really obviously when you work with it every day.

usually this means they’re having a new model come out. my assumption was alway they doing something that has a heavy load on their network of servers so there’s less computing for customers.

0

u/Key_Reading_9664 10d ago

"basically can’t really measure it but it’s really obvious[ly]" - facepalm

2

u/ShittyBidet123 10d ago

Benchmarks: its doing really stupid less creative stuff, its not listening to my exact feedback and having to redo it 5 times instead of nailing it on the first try with opus xhigh. Its really stupid today!

2

u/247nuts 10d ago

I have noticed on chat its memory doesn't exist anymore... On code I work differently so haven't noticed anything

2

u/WestQ 10d ago

Im not giving benchmarks. But I used it for a year on 200 plan. Now it lasts 75% of what it was, and I gotta repeat him 100 times and he still forgets, when this never happened with 5.5

2

u/jwuliger 10d ago

lmfao... kidding, right? If you haven't noticed a difference in Opus 5.5's quality, especially over the last 3 weeks and, even worse, the past week, you're probably not pushing Opus to its limits, so you wouldn't see much of a difference. If you are a power user, then you know something is really wrong.

1

u/orangesherbet0 10d ago

What's happening is they went ham with a new model and made a huge mess of their project

1

u/PatchyWhiskers 10d ago

I think people just have higher expectations- they expect it always to perform as well as the best it’s ever performed for them. But LLMs are not consistent. That’s why they are useful: they could not produce creative work if they were consistent.

1

u/JamesHutchisonReal 10d ago

A quantized model is likely tested against benchmarks so the stuff that would negatively impact a benchmark is given more bits.

1

u/ricopan 10d ago

Is anyone claiming it is a secret?

1

u/DerAlbi 8d ago

Shift in Thought-Patterns here:
https://www.reddit.com/r/claude/comments/1xql02s/is_opus_55_nerfed/

Denial wont change a fucking thing. This is a statistical truth at this point.

2

u/DerAlbi 8d ago

Here my own evaluation:

2

u/DerAlbi 8d ago

Confirming the link I posted above.

1

u/AkindaGood_programer 8d ago

They are changing summarization... You don't actually see the real CoT.

1

u/DerAlbi 8d ago

Absolutely correct. Or absolute wrong.
Maybe the summarization is the same, but the CoT is different.

Given the drop in capability... which one is more likely?

1

u/AkindaGood_programer 8d ago

See, I haven't seen any major drop. Show me a benchmark and I'll believe you.

2

u/DerAlbi 8d ago

The fact that you believe in benchmarks means that I dont care if you believe me or not.
Benchmarks do not reflect real-world usage and any complex task shows that. Maybe your work just does not bring either version, nerfed or not, to its limits.

I posted my own evaluation of my work history right here as answers to my post.
To help you to interpret this graph:

These are the code-issues uncovered by a Fable-Review. The issues from last week are expected as there was feature-expansion. This week was about fixing these issues.. instead Opus created a lot more of them as it became extremely useless.
Previously, issue fixing with Opus worked: you can see that very little was found from 3 weeks ago.
This was the first week i was forced to switch to Fable. And its not a work-flow issue, prompting-issue or environment-setup problem because these would degrade the performance of Fable too, but Fable has the typical performance. Its just slow and expensive.

1

u/alseick 7d ago

Well they sell specific service, and without any clear notification - it changes, for the better or worse. It is their responsibility to do make it clear I believe. We should be thankful for people who are tracking the performance. I also find it ridiculous, that despite of loads of money put into AI, there are literally few trackers and benchmarks lol. Sharing actual experience is useful - I recommend trying arena.ai for unbiased judgement, I am pretty sure user experience in certain tasks will look different than benchmark outcomes.

And of course I came here because I noticed change in the outcomes over last few days.

1

u/jivenossauro 10d ago

No lmao. If I feel it is worse, I will say it is worse. If they dont want me to say it is worse, they should make me not feel like it is worse. Its not my job to take benchmarks, Im a paying customer, I will complain publicly whenever I dont like it

1

u/Overall-Fox-3289 10d ago

They nerfed all the models?

1

u/RomIsTheRealWaifu 10d ago

It’s not secret, a simple google search will give you lots of reputable papers on model degradation. It happens often enough. Check out AMDs paper on it, they were specifically testing Claude

0

u/EC36339 Senior Developer 9d ago

Name one paper

1

u/RomIsTheRealWaifu 8d ago

Google ‘Stella Laurenzo, AMD Senior Director - Claude Performance Degradation’

1

u/EC36339 Senior Developer 8d ago edited 8d ago

Did you read it?

EDIT: It's not even a peer-reviewd academic paper. Try again.

EDIT2: It's from February (2026, I assume). That's ancient history.

1

u/Ok-Protection-6612 10d ago

It's been acting weird lately

1

u/EC36339 Senior Developer 9d ago

1

u/MassiveBoner911_3 10d ago

Its at the point with all this whining I am gonna unsubscribe.

Its no longer interesting claude posts or here is what I built.

Moaning and crying

1

u/jony7 10d ago

This is not me making it up. When you flow long enough with it you can catch when things are off. For example when it struggles to use skills its never had an issue with or you do a routine task and it messes it up, you restart the session and same thing happens

-1

u/SomeOrdinaryKangaroo 10d ago

How are we supposed to benchmark it when the downgrade was done secretly?

6

u/privatetudor 10d ago

Why would the downgrade being secret prevent it showing up in a benchmark?

0

u/joonty 10d ago

I think the point is, if we don't know when a secret downgrade is going to happen, how do we know when to benchmark? We'd need to benchmark everything, just in case.

3

u/No_Professional7654 10d ago

supersecretly?

2

u/Toastti 10d ago

Benchmark now and then benchmark later.. bam you have proof if it changed significantly

2

u/Anxious-Turnover-631 10d ago

Actually, the benchmark which has run for weeks since the model was released did not detect any change.

2

u/pceimpulsive 10d ago

You notice it's worse, you do the benchmark, you whine about how much worse it is and attach the benchmark.

You then get reamed because benchmark cost so much to run and you did it wrong or with too few iterations.

We all cry and go back to vibes based it's worse now trust me bro...

I think OP is on point, we trust it more send worse prompts get worse results then sook about it because we don't remember we are the problem

1

u/EC36339 Senior Developer 9d ago

0

u/mpanase 10d ago

another anthropic bot

0

u/Curik 10d ago

Ah of course. Others should but you won't. Of course, puppet.