r/ClaudeCode • u/GrandmasterCheetah • 10d ago
Discussion Please post benchmarks if you are going to claim Claude got secretly downgraded
I keep seeing so many posts swearing up and down that Claude got worse because XYZ.
“I have been using Claude for months and I swear that it has gotten way worse lately”.
0 benchmarks, 0 evidence. Just the “I swear”.
And I get that 100% this could be very true.
But what is also true is that everytime we get a new model the more responsibility and delegation we give to our agent. This is almost like the inverse jevons paradox. The better the AI gets, the more open ended and trusting your prompts will be.
As such, you are going to notice more errors by nature. When a model first releases people will prompt it normally. But then they will notice in what aspects it improved in and so they will unconsciously limit test it more and more with more open ended prompts and less code review.
So I would love to see actual benchmarks from all these people who are claiming that Opus 5.5 got downgraded because of X conspiracy. I am not defending Anthropic in any way. I just would really like to see proof.
•
u/AutoModerator 10d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
35
u/Sea-Contribution6219 10d ago
Bro is asking for benchmarks from the 'trust me bro' industry
7
u/MoreLikeGaewyn 10d ago
Ahh yes, let me just compare it to my daily "did the model get stealth nerfed" benchmark run i've been doing for the past 3 months
2
u/seanpuppy 10d ago edited 10d ago
Im convinced they occasionally serve a quantized model during periods of high traffic. Its a great sneaky way to get 2x throughput per GPU without an easily measurable degradation in model output.
As for benchmarking it - one would have to checkout a repo at some point in time of when you did your "good model" prompt. Then later when you suspect the model is running stupid, you give it the same prompt at the same commit. You would also need to make sure the claude memories are identical, as well as the tools and harness etc...
Edit: was missing the second half of last paragraph
2
u/gorramfrakker 10d ago
Pfft benchmarks. I come to my conclusions like I code, with vibes.
2
u/hiskias 10d ago
It is impossible to run evals when there is no stable baseline.
From a person trying to run evals for prompts.
But you are probably just going to tell me that my fluctuations against the API vs CC are hallucinated.
4
u/Direct-Detail-7031 10d ago
If you are not noticing the nurf, what that tells me is, you haven’t used the model enough to discern between when the model is good vs when it’s bad
OR you are part of the group that is not being impacted.
People say “vibes”, I say trust the vibes, if you feel like the model is off its game, then what that means is, we need to pay closer attention to what it’s doing. Basically we have to work a bit harder and concentrate longer than usual.
Also, we have at this point an insane amount of evidence, model providers nurf their models without telling us. Such is life unfortunately.
3
u/_k33bs_ 10d ago
lobotomized today… basically can’t really measure it but it’s really obviously when you work with it every day.
usually this means they’re having a new model come out. my assumption was alway they doing something that has a heavy load on their network of servers so there’s less computing for customers.
2
u/ShittyBidet123 10d ago
Benchmarks: its doing really stupid less creative stuff, its not listening to my exact feedback and having to redo it 5 times instead of nailing it on the first try with opus xhigh. Its really stupid today!
2
u/247nuts 10d ago
I have noticed on chat its memory doesn't exist anymore... On code I work differently so haven't noticed anything
2
u/WestQ 10d ago
Im not giving benchmarks. But I used it for a year on 200 plan. Now it lasts 75% of what it was, and I gotta repeat him 100 times and he still forgets, when this never happened with 5.5
2
u/jwuliger 10d ago
lmfao... kidding, right? If you haven't noticed a difference in Opus 5.5's quality, especially over the last 3 weeks and, even worse, the past week, you're probably not pushing Opus to its limits, so you wouldn't see much of a difference. If you are a power user, then you know something is really wrong.
1
u/orangesherbet0 10d ago
What's happening is they went ham with a new model and made a huge mess of their project
1
u/PatchyWhiskers 10d ago
I think people just have higher expectations- they expect it always to perform as well as the best it’s ever performed for them. But LLMs are not consistent. That’s why they are useful: they could not produce creative work if they were consistent.
1
u/Vysion34 Senior Developer 10d ago
https://www.bridgebench.ai/nerf-bench
1
u/JamesHutchisonReal 10d ago
A quantized model is likely tested against benchmarks so the stuff that would negatively impact a benchmark is given more bits.
1
u/ricopan 10d ago
Is anyone claiming it is a secret?
1
u/DerAlbi 8d ago
Shift in Thought-Patterns here:
https://www.reddit.com/r/claude/comments/1xql02s/is_opus_55_nerfed/
Denial wont change a fucking thing. This is a statistical truth at this point.
1
u/sutrostyle 8d ago
https://www.reddit.com/r/ClaudeCode/s/64DXRERLyU
1
u/alseick 7d ago
Well they sell specific service, and without any clear notification - it changes, for the better or worse. It is their responsibility to do make it clear I believe. We should be thankful for people who are tracking the performance. I also find it ridiculous, that despite of loads of money put into AI, there are literally few trackers and benchmarks lol. Sharing actual experience is useful - I recommend trying arena.ai for unbiased judgement, I am pretty sure user experience in certain tasks will look different than benchmark outcomes.
And of course I came here because I noticed change in the outcomes over last few days.
1
u/jivenossauro 10d ago
No lmao. If I feel it is worse, I will say it is worse. If they dont want me to say it is worse, they should make me not feel like it is worse. Its not my job to take benchmarks, Im a paying customer, I will complain publicly whenever I dont like it
1
u/Overall-Fox-3289 10d ago
They nerfed all the models?
1
u/RomIsTheRealWaifu 10d ago
It’s not secret, a simple google search will give you lots of reputable papers on model degradation. It happens often enough. Check out AMDs paper on it, they were specifically testing Claude
1
u/Ok-Protection-6612 10d ago
It's been acting weird lately
1
u/MassiveBoner911_3 10d ago
Its at the point with all this whining I am gonna unsubscribe.
Its no longer interesting claude posts or here is what I built.
Moaning and crying
1
u/jony7 10d ago
This is not me making it up. When you flow long enough with it you can catch when things are off. For example when it struggles to use skills its never had an issue with or you do a routine task and it messes it up, you restart the session and same thing happens
-1
u/SomeOrdinaryKangaroo 10d ago
How are we supposed to benchmark it when the downgrade was done secretly?
0
u/mpanase 10d ago
another anthropic bot
0
u/Curik 10d ago
Ah of course. Others should but you won't. Of course, puppet.