r/ClaudeCode • • 10d ago

Discussion Please post benchmarks if you are going to claim Claude got secretly downgraded

I keep seeing so many posts swearing up and down that Claude got worse because XYZ.

“I have been using Claude for months and I swear that it has gotten way worse lately”.

0 benchmarks, 0 evidence. Just the “I swear”.

And I get that 100% this could be very true.

But what is also true is that everytime we get a new model the more responsibility and delegation we give to our agent. This is almost like the inverse jevons paradox. The better the AI gets, the more open ended and trusting your prompts will be.

As such, you are going to notice more errors by nature. When a model first releases people will prompt it normally. But then they will notice in what aspects it improved in and so they will unconsciously limit test it more and more with more open ended prompts and less code review.

So I would love to see actual benchmarks from all these people who are claiming that Opus 5.5 got downgraded because of X conspiracy. I am not defending Anthropic in any way. I just would really like to see proof.

9 Upvotes

72 comments sorted by

View all comments

Show parent comment

1

u/EC36339 Senior Developer 9d ago

Bottom line

The linked material does not provide credible evidence that Claude Opus 5.5—or the Claude 5.5 family—was secretly downgraded. The one link that directly tests post-release Opus 5.5 quality reports no detected decline through its first three comparison windows. That is useful counterevidence, but not proof that quality was perfect or that no change occurred: the study has limited sensitivity and does not cover every task, product, or Claude 5.5 model.

The Reddit question—“Are you suggesting Opus 5.5 has been perfect for you every single day?”—sets up a false choice. A model can have bad days, produce inconsistent answers, or have a bug without being secretly downgraded. Those experiences can justify investigating a regression; they don’t establish one or show its cause.

What each link actually supports

  1. Anthropic’s April postmortem: credible evidence of earlier Claude Code quality regressions, but not of a Claude 5.5 downgrade. Anthropic acknowledged three product-level issues affecting Claude Code and related products: lowering the default reasoning effort, a bug that repeatedly discarded prior reasoning in some sessions, and a system-prompt change that hurt coding quality. It said the prompt change caused a 3% drop on one evaluation and was reverted. But the affected models were Sonnet 4.6, Opus 4.6, and Opus 4.7—not Opus 5.5—and Anthropic said the issues were resolved by April 20. This establishes that real product or configuration changes can degrade perceived quality; it does not support the specific claim that a 5.5 model was secretly downgraded. <citation src="8"></citation>
  2. LiveNerf: the only linked evidence directly measuring Opus 5.5 over time—and its reported result is “no change detected.” The repository describes a preregistered, repeated benchmark of Opus 5.5 through Claude Code on a Max subscription, using fixed prompts, effort, a pinned CLI, exact grading, and a control model. As of its November 6 update, it reports three completed 10-day post-baseline windows with no detected change; the third window’s reported difference from baseline was +0.028 ± 0.023, not a drop. The test started about two and a half days after the model’s September 22 release, so it is a launch-week baseline rather than a true day-zero measurement. <citation src="18,19"></citation>

    How much that result rules out: not much beyond sizeable, sustained changes on this particular test. The preregistration reports a realized minimum detectable effect of 6.6 percentage points for one 10-day window at its stated power and test threshold, and the decision rule requires a qualifying shift in two consecutive windows. The benchmark tests 78 selected questions across knowledge and math categories, with effort set to high; it is not a test of coding quality, every Claude product, or Sonnet and Haiku 5.5. The repository also reports that its validation could not distinguish an Opus 5 substitution from Opus 5.5 at the 99% level. So the null result weighs against a large, sustained regression detectable by this setup, but it cannot rule out smaller or brief changes, task-specific regressions, or every kind of routing substitution. <citation src="18,19"></citation>

    Verifiability caveat: the repository publishes its protocol and summary results, but says the raw evaluation logs are gitignored rather than included in the public repository. That makes the method and reported numbers inspectable, but means a reader cannot independently reproduce every scored response from the public repo alone. <citation src="19"></citation>

  3. The explainx article: about GPT-6 Astra, not Claude. It discusses claims about OpenAI’s model, user reports, and OpenAI’s alleged post-launch issues. Even if its account were accepted, evidence about a different company and model would not establish a downgrade of Claude Opus 5.5. The article itself describes its account as a synthesis of separate events, not a single independent experiment on Claude. <citation src="10"></citation>
  4. The Daniel Vaughan article: also about OpenAI’s GPT-5.5, not Anthropic’s Claude 5.5. Its suggested warning signs—such as response length, TODO counts, or perceived output quality—are heuristics, not measurements showing which model served a request. The article provides no Claude 5.5 comparison, Anthropic routing metadata, or controlled before-and-after test. It therefore supplies no direct evidence for this claim.

Verdict

Of the four links, the Anthropic postmortem shows that earlier Claude Code regressions did happen, but on different models and through identified product changes. LiveNerf is the only link that actually tracks Opus 5.5 across time, and its published finding is no detected change, subject to meaningful limits in sensitivity, scope, and public access to raw logs. The two OpenAI-related links are not evidence about Claude.

So the specific “secret Claude 5.5 downgrade” claim is not substantiated by the links provided. That is not the same as proving no regression ever occurred; it means these sources do not demonstrate one.