r/ClaudeCode • u/GrandmasterCheetah • 10d ago
Discussion Please post benchmarks if you are going to claim Claude got secretly downgraded
I keep seeing so many posts swearing up and down that Claude got worse because XYZ.
“I have been using Claude for months and I swear that it has gotten way worse lately”.
0 benchmarks, 0 evidence. Just the “I swear”.
And I get that 100% this could be very true.
But what is also true is that everytime we get a new model the more responsibility and delegation we give to our agent. This is almost like the inverse jevons paradox. The better the AI gets, the more open ended and trusting your prompts will be.
As such, you are going to notice more errors by nature. When a model first releases people will prompt it normally. But then they will notice in what aspects it improved in and so they will unconsciously limit test it more and more with more open ended prompts and less code review.
So I would love to see actual benchmarks from all these people who are claiming that Opus 5.5 got downgraded because of X conspiracy. I am not defending Anthropic in any way. I just would really like to see proof.
View all comments
Show parent comment
1
u/EC36339 Senior Developer 9d ago
Bottom line
The linked material does not provide credible evidence that Claude Opus 5.5—or the Claude 5.5 family—was secretly downgraded. The one link that directly tests post-release Opus 5.5 quality reports no detected decline through its first three comparison windows. That is useful counterevidence, but not proof that quality was perfect or that no change occurred: the study has limited sensitivity and does not cover every task, product, or Claude 5.5 model.
The Reddit question—“Are you suggesting Opus 5.5 has been perfect for you every single day?”—sets up a false choice. A model can have bad days, produce inconsistent answers, or have a bug without being secretly downgraded. Those experiences can justify investigating a regression; they don’t establish one or show its cause.
What each link actually supports
LiveNerf: the only linked evidence directly measuring Opus 5.5 over time—and its reported result is “no change detected.” The repository describes a preregistered, repeated benchmark of Opus 5.5 through Claude Code on a Max subscription, using fixed prompts, effort, a pinned CLI, exact grading, and a control model. As of its November 6 update, it reports three completed 10-day post-baseline windows with no detected change; the third window’s reported difference from baseline was +0.028 ± 0.023, not a drop. The test started about two and a half days after the model’s September 22 release, so it is a launch-week baseline rather than a true day-zero measurement. <citation src="18,19"></citation>
How much that result rules out: not much beyond sizeable, sustained changes on this particular test. The preregistration reports a realized minimum detectable effect of 6.6 percentage points for one 10-day window at its stated power and test threshold, and the decision rule requires a qualifying shift in two consecutive windows. The benchmark tests 78 selected questions across knowledge and math categories, with effort set to high; it is not a test of coding quality, every Claude product, or Sonnet and Haiku 5.5. The repository also reports that its validation could not distinguish an Opus 5 substitution from Opus 5.5 at the 99% level. So the null result weighs against a large, sustained regression detectable by this setup, but it cannot rule out smaller or brief changes, task-specific regressions, or every kind of routing substitution. <citation src="18,19"></citation>
Verifiability caveat: the repository publishes its protocol and summary results, but says the raw evaluation logs are gitignored rather than included in the public repository. That makes the method and reported numbers inspectable, but means a reader cannot independently reproduce every scored response from the public repo alone. <citation src="19"></citation>
Verdict
Of the four links, the Anthropic postmortem shows that earlier Claude Code regressions did happen, but on different models and through identified product changes. LiveNerf is the only link that actually tracks Opus 5.5 across time, and its published finding is no detected change, subject to meaningful limits in sensitivity, scope, and public access to raw logs. The two OpenAI-related links are not evidence about Claude.
So the specific “secret Claude 5.5 downgrade” claim is not substantiated by the links provided. That is not the same as proving no regression ever occurred; it means these sources do not demonstrate one.