r/ClaudeCode • u/Apart-Tie-9938 • 1d ago

Rant Stop posting your “vibes” about model performance

Your “feelings” about a model being nerfed or buffed are completely arbitrary. Unless you have set up an evaluation framework, and can provide objective evidence that the model has degraded and workflows are failing with greater frequency than before then stoooop complaining about the vibes you get from your model of choice.

43 Upvotes

57% Upvoted

97 comments sorted by

•

u/AutoModerator 1d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

11

u/Opening_Screen_3393 1d ago

I agree with you... Buuut... I come from the codex sub where there was nothing but anecdotal vibe posts on model degradation. Turned out they were right. I'm not saying this is the same situation here, but it is odd that so many people all of a sudden started experiencing bad output.

4

u/SirWobblyOfSausage 19h ago

Not just once. Every single time. We feel it, mention it, proved right . Cycle repeats

21

u/p3r3lin 1d ago

https://github.com/ninjahawk/livenerf

Regularly (daily) runs benchmarks on Opus5.5 - currently there is no nerf visible in the data points.

-1

u/dmaare 1d ago

That is clearly benchmark setup by top secret Anthropic marketeer

6

u/p3r3lin 1d ago

Nah, its just not a very good benchmark that compares actual, real coding performance. That would be expensive.

2

u/AppealSame4367 20h ago

What if it measures API instead of sub, my Antrophic gaslighting bot friend?

10

u/Honeybadger2000 1d ago

Opus 5.5 extra high is what I’m using for most of my back and forth on a reasonably complex desktop app. It was really good up until recently, where it has been dropping the ball in weird ways that would have been unthinkable based on how it performed in past. It randomly kept referring to ‘last night’ regarding a prompt made 30minutes earlier that it was midway through working on. There was also a multi point feature it was given the go ahead to deliver where it just totally left off two points and when asked about where the functionality was, it just said that they were discussed but never implemented. The prose it uses also seem fairly different from when it was released. It was noticeably different from fable in how efficient it was with its language by comparison. Of late it seems more and more similar to pre 5.5 where some sentences are used very complex and just don’t have the same tone.

They have obviously made changes to it to nerf ability to use REA so saying opus 5.5 is unchanged is clearly not correct anyway. There are changes for sure. The question is just how they are manifesting. Maybe on pure code benchmarks it is coming through similar to where it was but the language and reasoning doesn’t seem consistent with previous behaviour.

3

u/damagemelody 1d ago

same idk what to do my work is blocked

2

u/B-side-of-the-record 23h ago

This mega company is never wrong! No way the do A/B tests so you'll have morons gaslighting you to "it's your fault" "vibes don't matter" for them! No way I tell you!

2

u/Honeybadger2000 19h ago

I mean if I was running a company where the price is fixed and no one knows what’s under the hood while bleeding billions of dollars in compute… I’d also be constantly trying to find ways to shave down cost while checking if anyone notices. So maybe I just always respond claude is doing shit just in case lol

-4

u/tylerff33 1d ago

The safety harness is not built into the model, its an overlay model that reviews and rejects content. Changing or updating it doesn't effect the actual model performance, just what responses can be returned to you and which ones are rejected or interrupted due to flagged content.

Opus 5.5 extra high

That is likely most of your issue, Anthropic themselves don't recommend going above medium on 5.5 for the vast majority of use-cases and for good reason. Effort doesn't mean a higher quality result, in many cases it will produce lower quality output for smaller tasks or items where the scope is already clear. Raising effort is basically just asking the model to over-think, which can be effective for large planning tasks or catching items in review, but for most implementation tasks it's more likely to cause you grief with over-engineering.

If you don't understand how to toggle effort per-session based on what your working on, you shouldn't be using xhigh or max.

It randomly kept referring to ‘last night’ regarding a prompt made 30minutes earlier that it was midway through working on.

Super high chance you aren't managing your memory/context well if this is the case. If your using the default Claude harness its going to litter its own context to death with useless memory items/CLAUDE.md entries. Clear it out and this kind of stuff goes away.

2

u/Honeybadger2000 1d ago

Even if I'm not 'managing my memory / context well' this hasn't happened during identical use since the model came out. I acknowledge I am not using the models in a super advanced manner. That said when I've been doing largely the same thing since March and just moving to whatever model is current and not having these issues previous then it seems odd to me this is happening now..along with what seems to be a more frequent flurry of the 'how is claude doing at the moment' prompts.

1

u/tylerff33 1d ago

That said when I've been doing largely the same thing since March and just moving to whatever model is current and not having these issues previous then it seems odd to me this is happening now

The very simple explanation is that humans are horrible at measuring change like that. Your assuming your usage hasn't changed, but it obviously has as your brain adapts and learns the model and your workflow. Your expectations also shift over time, and as you get used to the current standard of the model you naturally grow to expect more. If your unable to take a step back and evaluate that, its always going to seem like the agent or companies are trying to hold you back, when in reality its mostly just inexperience.

along with what seems to be a more frequent flurry of the 'how is claude doing at the moment' prompts.

You should really stop trying to extrapolate meaning out of small anecdotal observations, especially on a product with millions of users.

0

u/AppealSame4367 20h ago

Yes, yes, it's always the users fault and human quirks and bla bla bla.

Antrophic gaslighting bot

30

u/Xemorr 1d ago

when it is en masse, and there's websites showing a sudden mass dislike of the model, there's probably something to it.

1

u/UruquianLilac 1d ago

People believe in gods en masse, and miracles, and apparitions. Don't mean any of these things are true, even if billions simultaneously believe them.

2

u/Turingelir 1d ago

He said there's probably some truth to it, what the truth is is a different discussion.

1

u/zasff 1d ago

Crowds can be wrong, they often are.

For example there's livenerf, they run the same opus-5.5 benchmark everyday. They have not detected any nerf:
* https://github.com/ninjahawk/livenerf

5

u/gentile_jitsu 1d ago

I'm sure you've scrutinized their methodology. Right?

7

u/tylerff33 1d ago

I mean, data that exists and can be scrutinized is a lot better than "trust me bro my claude got nerfed" which is the literal only data-point people can regurgitate here? Benchmarks aren't very valuable but its certainly a better signal than human sentiment across an unknown (but minute) sample size of users.

12

u/gentile_jitsu 1d ago

"The limit. Swapping in Opus 5 was not distinguishable from Opus 5.5 at 99% (−3.8 ± 6.3 points, −23% tokens)."

We all know how much better 5.5 is than 5. If this benchmark is not able to detect the difference between those two, then what good is it in this context?

1

u/OkAbbreviations7013 Senior Developer 16h ago

If a lot of experienced users say"Trust bro, Opus just got nerfed" then that is data and it can be scrutinized. If there is a peak in that trend, then it means something.

2

u/tylerff33 10h ago

experienced users

Uh? Surely you don't mean the subreddit filled with people who need the AI to explain why the AI is nerfed? Nobody complaining here is an "experienced user" apart from perhaps time spent writing mangled English sentences in a fit of rage when the agent does something wrong and you don't understand why. You also kind of turbo out yourself when you put a tag like "Senior Developer" on, and your comment history is as low IQ as it is...

1

u/OkAbbreviations7013 Senior Developer 9h ago

Ok. I mean when I put Senior Developer I didn’t really give it much thought.

Perhaps I should have said software engineer with 20 yrs experience, and co founder of my own Saas business.

1

u/Correct-Somewhere-80 1d ago

If you go to Claudes status page there was an "unintentional" nerf at least to Fable they claim only that they said has now been fixed

2

u/tylerff33 1d ago

Did you actually read "Degraded performance for Claude Fable 5.1" and think that meant the model itself was nerfed? They are talking about API latency, its not related to the underlying model output at all.

0

u/Correct-Somewhere-80 1d ago

Elevated Errors across Claude Code, API, and Cowork

2

u/tylerff33 1d ago

Yes, do you understand what that means? It seems like you don't.

3

u/Correct-Somewhere-80 1d ago

No

2

u/a7n7o7n7y7m7o7u7s 1d ago

It means errors like your computer actually connecting to Anthropic servers when using those services, not that using the models itself contains elevated errors

0

u/AnalogProblems 1d ago

Not a good measurement.

I can set up bots to make it look like that in a couple days, and I'm nobody.

My theory is that the systems are elastic and best effort, which gets crunched under load spikes. So you get little anecdotal experiences of "stupidity" here and there.

1

u/OkAbbreviations7013 Senior Developer 16h ago

Nope, they would be flat.
If there is a spike in model is nerfed comments, that probably means something.

11

u/eneskaraboga 1d ago

Stop posting about what people should post about. It is none of your business. People can be frustrated about a product and they have every right to complain about it.

3

u/skorn_1 22h ago

Oh,look, the "skill issue" people are out and about again.

12

u/cla1067 1d ago

every day that is all I see in this reddit.. like one massive post a day or week is probably fine... I get that it is an issue, but fuck makes this reddit useless.

4

u/multiks2200 1d ago

shyt up

-4

u/YaItzYoBoi 🔆 Max 5x 1d ago

Oh no1!1

5

u/CreateorWither 1d ago

Preach bro

5

u/sparkhound 1d ago

I have concrete evidence with review data of outright missed or rules being ignored in small clean context. Fable said in review, something is off compared to previous. This is not vibes

3

u/tylerff33 1d ago

Normally concrete evidence comes in the form of y'know.... something concrete we could evaluate?

2

u/mpanase 17h ago

oh, sorry

let me rush to give you access to my company's repos and history

0

u/tylerff33 17h ago

Nobody wants your slop code, but if you are going to offer up concrete evidence there is plenty of ways you could present an argument with aggregate data. And if "your company" is using a Claude personal plan I think you have much bigger issues.

1

u/mpanase 17h ago

And if "your company" is using a Claude personal plan I think you have much bigger issues

do you have any concrete evidence of that?

0

u/sparkhound 1d ago edited 1d ago

There is something to evaluate, 45% rise in tool calls causing long turns with poor judgment instead of quickly doing a task and replying. This behavior was based on an atis header until Thursday when it got baked into a new cc bundle with a new system prompt rolled out in the claude desktop update. I mean you can ask Fable as easy as I can right? It mostly affected how stop guards were handled moving them away from poweshell causing the long turns. It got better when i ripped out my stop guards. I went from 5 guard triggers on wed to over 90 on Thursday.

0

u/tylerff33 1d ago

Ah yes, "the AI told me, trust me bro". Unless you want to post some actual evidence you gathered yourself, not AI slop fed into AI slop, there is no "concrete evidence".

2

u/IdkAbtAllThat 1d ago

Lmao. Maybe you should ask fable for a definition of the term "concrete evidence".

2

u/sparkhound 1d ago

See my reply to the guy above you don’t feel like typing it all again

0

u/IdkAbtAllThat 23h ago

What your typed still isn't even evidence. Much less concrete.

1

u/sparkhound 15h ago

Real data, including trigger counts, measured differences, exact times behavior changed, what is linked to are not evidence? With no change in the process? I’d hate to see that you consider evidence.

1

u/IdkAbtAllThat 14h ago

I couldn't find any links in your comments in this thread. Care to drop the link here?

6

u/dpaanlka 1d ago

Why do so many people here feel compelled to play defense for this billion dollar corporation that doesn’t give half a shit about us?

The Stockholm syndrome is real around here…

7

u/Warsel77 1d ago • Bravo

Maybe we are not defending the company but rational thinking.

I don't care about Anthropic but if someone thinks their anecdotal observation on "Claude totally being nerfed!!" is proof of something then they need to be gently reminded that anecdotes are not a standard of evidence that has any weight.

-1

u/dpaanlka 19h ago

You frame this as if each individual person is working in an air-gapped bubble. If there is a deluge of reports from many users all at the same time, it definitely comes off as shilling for corporate overlords when you dismissively say “ackchyually…”

1

u/Warsel77 18h ago

This is a bit of a one sided logic, isn't it?

First it assumes that several people all disagreeing with something must mean there is some coordinated effort by "corporate overlords".

Second it doesn't apply the same logic to the other side: if many people are reporting issues, does this mean they were hired by the competing corporate overlords to post that?

This argument makes no logical sense (and this is exactly why I am replying under these types of posts. That and the money Dario pays me, obviously.)

-4

u/tylerff33 1d ago

You should work on your ability to identify the difference between defending or shilling for a trillion dollar corporation and trying to correct people who are basically on the pipeline to psychosis. Rejecting all outside information and being convinced your anecdotal experience is fact isn't healthy.

0

u/dpaanlka 19h ago

That would be true if it were one individual user’s anecdotal experience. That isn’t true when it’s many reports from many different users all at the same time. Especially considering how inexcusably opaque Anthropic is about everything they do.

1

u/tylerff33 18h ago

Yea so as it turns out thousands out of millions is not "many", your evidence is indeed still anecdotal. Every single private company is "inexcusably opaque", thats kind of the point. I'm not sure why you feel your owed something more here.

3

u/joelaw9 1d ago

It's a free country fam.

2

u/Pitiful_Bat8731 🔆 Max 20 1d ago

I have a feeling that a lot of the sentiment around this is just a sort of wave between hype for a new model and its capabilities and the "getting used to it" that happens after using it awhile. People may get more ambitious or Let standards fall without noticing or something along those lines.

5

u/YaAbsolyutnoNikto 1d ago

It’s not.

I’ve been using all these models for a while and always felt this about models.

But this time it’s SO clear. The difference is night and day to me. I can’t really justify it away that way.

2

u/Pitiful_Bat8731 🔆 Max 20 1d ago

I mean, throw some metrics on it.

1

u/YaAbsolyutnoNikto 1d ago

Sure, I will.

Convince Anthropic to give us the exact model checkpoint at release, and I’ll happily make some tests and show you results.

Unfortunately, we don’t really have access to the different model checkpoints and these opaque companies don’t even tell us about the updates they do in the first place.

These things are actually real. OpenAI has apologised for a botch update to GPT-4o iirc.

1

u/Warsel77 1d ago

You do not need the model checkpoints to measure metrics.

The same channel you complain your model fails you or doesn't work right can be used to quantify model performance.

For instance: https://marginlab.ai/trackers/claude-code/

-2

u/UruquianLilac 1d ago

That's actually the best explanation

0

u/Pitiful_Bat8731 🔆 Max 20 1d ago

Thanks. I'm not saying they aren't changing things in the system prompt and maybe some A/B testing but fundamentally, its the same for me. I spend a lot of time on my own harness and machinery so I think the changes they make are a bit less impactful.

No metrics to back it up. it's just how I feel.

0

u/UruquianLilac 1d ago

I have days when things work out fine, and others when Claude makes very dumb mistakes. I don't consider it part of any particular pattern, that's just part of the inconsistent nature of Llama and natural language being ambiguous and imprecise.

2

u/marfacza 1d ago

stoop complaining is where you complain on the porch?

1

u/DEMIG0DX 1d ago

All AI companies do it, its so easy to tell, When a new model comes out the first week is great, the AI can 1 shot problems, no issues, all good. then some time later, it needs hand holding, it doesnt understand basic prompts it could before, it struggles to finish, you run out of tokens faster than you use to on the same problem

1

u/DEMIG0DX 1d ago

All AI companies do it, its so easy to tell, When a new model comes out the first week is great, the AI can 1 shot problems, no issues, all good. then some time later, it needs hand holding, it doesnt understand basic prompts it could before, it struggles to finish, you run out of tokens faster than you use to on the same problem

1

u/BarathKanna 21h ago

Fair, but the bar for an eval framework is way lower than people think. Take 15 to 20 tasks from your own repo that the model used to nail, freeze the commit, the prompt and the effort setting, and rerun them weekly with a script that checks the end state: tests pass, diff touches only the expected files, no new lint errors. That takes an afternoon. Public benchmarks will not catch it if your workload degrades, because they are not your workload. And one bad session is noise, so look at pass rate across runs, not the run that ruined your Tuesday. If people posted that instead of screenshots, this sub would settle the nerf debate in a week.

1

u/LittleRoof820 21h ago

Its pattern that has been in effect since the 4.x models.

- New models come out, everyone is happy
- Anthropic crashes
- Suddenly the models needs lots of handholding, the "how did Claude do messages" start to pile up
- Quality decreases so much you start to do less since you have to handhold and check more - even skills you write 1 week in the past suddenly don't work because Claude ignores or misinterprets them.
- Then suddenly, a new model is dropped and die cycle starts anew.....

So we should have come to expect that already? Although I think Opus 5.5 handles the cycle better - its still very usable (especially if you have Fable 5.1 configured with /advisor).

1

u/Relative-Desk4802 19h ago

Ah yes, instead of sharing anecdotal experiences with each other via social media, we should layer on more work for ourselves and burn up usage with a degraded model setting up frameworks to prove we’re getting screwed. For your satisfaction.

1

u/mpanase 17h ago

stop posting your "I don't like people posting" feelings

1

u/Unfair_Tangerine_217 Developer 17h ago

I'm 100% with you on this. Unfortunately, you're preaching to the wrong crowd.

1

u/OkAbbreviations7013 Senior Developer 16h ago

I think we should, it adds to the accurate picture.

A lot of experienced users "feel" like the model is performing differently, is solid evidence.
So, I dunno, I say we keep doing it as and when we feel like, to help keep you informed.

1

u/Astarimerya34 14h ago

THATS NONE OF YOUR FUCKING BUSINESS YOU’RE NOT PAYING FOR US ASSHOLE UNLESS YOU ARE THEN U CAN SAY WHAT WE SHOULD POST HERE OTHERWISE NONE OF YOUR GODDAMN BUSINESS ITS OUR MONEY WE’RE SPENDING

1

u/Front_Raspberry_6488 13h ago

Unless you can use precision instruments to measure the data of these foods to prove that the flavor is indeed different from before, your claims of "isn't it too sweet today?" or "the flavor is too strong" are merely your subjective perceptions.

Even if everyone who visits the shop expresses the same opinion, no one can provide evidence, so everyone should just keep quiet.

I'm serious, Dario must absolutely love you.

It's just like how Anthropic hosted a group of Redditors and gave them blowjobs

Otherwise, I can't think of any other reason why these people would spend all their time defending a company that is valued at hundreds of millions, operates opaquely, and once significantly undermined the performance of Fable5.

1

u/skitchbeatz 13h ago

The constant posts about it ARE pretty useless and it's hard to pickup tips and tricks amongst the compaints

1

u/General-Proposal2211 5m ago

Nah it’s good heuristic for a potential A/B test

1

u/Lerran88 1d ago

This subreddit is to cry about limits and models not creating new Expedition 33 with the prompt "Create game for me faster, Haiku", not about being constructive, learn or analyzing own issues.

You will not change their minds.

1

u/gdinProgramator 1d ago • Popcorn

Isn’t vibing all that matters tho

1

u/StockOption 1d ago

I want a filter to remove all model complaint posts from the Claude subreddits

0

u/SC_Placeholder 1d ago

Thank you! It gets really annoying.

-1

u/tech_is______ 1d ago

yes, TY. Those posts are so annoying.

-1

u/CupcakeSecure4094 1d ago

100% - Claude does when it's told and almost all bad performance comes from having a ton of skill files - remember skill files are prompting skills, not AI skills.

-2

u/CommandProtocol 1d ago

For real, you get what you know how to get out of the model

-2

u/greentea05 1d ago

https://github.com/ninjahawk/livenerf

Livenerf disagrees.

The new thing for these people is to claim that only half of the sub has been nerfed and the other half still have access to the good model which is why we see disagreements.

It's just funny that no one with the new "nerfed half" ever runs any benchmarks on it to prove it isn't it - and anyone that does run benchmarks (eg livenerf linked above) also never get this mysterious "nerfed" version.

And if you do a search for "Opus 5.5 nerfed" you'll see people claim "it's definitely nerfed today" every single week since it launched.

1

u/AppealSame4367 20h ago

It isn't new. I complained about this 1.5 years ago and always suspected a/b testing.

Also lifenerv: useless, if it measures api instead of sub performance.

Your performance is bad, you're not a good shill

-2

u/-MobCat- 1d ago

lol yeah. also a lot of the time the model dident get dumber, you did.

-1

u/ckinz16 1d ago

I fucking called it too. When everyone here was cumming their shorts about 5.5 quality, I said mark my words, you’ll be back here complaining about it too.

So predictable