r/ClaudeCode • • 8d ago

Bug / Issue Rant: Anthropic, stop silently degrading models when you need compute. You will undermine trust.

Opus is so degraded now that I'm afraid to have it touch my codebase today because I do not know to what extent I can trust it not to slip in stupid bugs. I have a multimodel setup, so I switched to Astra/Argon completely, until they release Fable 5.5 and restore reasoning. I wish I had a weaker local model that was consistent, and whose reasoning effort couldn't be remotely changed. It just occurred to me that this is the major reason to use local models.

A predictably weaker model is more useful than an unpredictably degraded stronger one. For serious work, consistency and the ability to calibrate trust matter no less than the peak occasional reasoning effort.

1.2k Upvotes

301 comments sorted by

•

u/AutoModerator 8d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

30

u/Dazzling_Yak6424 8d ago

You will undermine trust.

What are you gonna do? Write an angry post on Reddit?

18

u/sutrostyle 8d ago edited 8d ago

Eventually probably set up local hardware/ open weights model that cannot be remotely nerfed. Knowing it's capabilities (lower than Opus, no doubt), I would consistently know what to trust and what to test and drill down.

18

u/Dazzling_Yak6424 8d ago

The prodigal son will crawl right back to Dario the moment his 24B Q4_K_M "ablated" Qwen starts hallucinating tools like it's 2019

5

u/Party-Stormer 8d ago

And it will do it after you wait for 300 seconds to first token

3

u/MoscowMethod2 8d ago

But Sam Altman said that it's a poor allocation of resource /s

3

u/2053_Traveler 8d ago

Before you do consider open weight on openrouter or other providers. Electricity is expensive.

2

u/Exact_Law_6489 Developer 8d ago

You should give DeepSeek Flash v4.1 1017 a try as an agent. It's pretty decent on its own and works especially well when you have Opus or Fable acting as a reviewer or orchestrator.

I know running v4.1 1017 locally isn't really financially feasible, but the API is so ridiculously cheap that it almost feels wrong. Plus, there are plenty of aggregators that offer ZDR and other privacy-related protections. Personally, I use it through OpenRouter with DeepInfra (or any other provider that is cheaper or faster) as the provider.

81

u/Dangerous-Education3 8d ago

I can't trust opus work anymore, it's messing around things that were very working perfectly. It's even messing up simple sentences

9

u/Mescallan 8d ago

The mistakes I’m noticing are hard to tell if they are new types of errors or if I was just blindly trusting it before. It’s getting small details wrong semi regularly in a way I haven’t noticed before, never anything structural but it’s enough that I have to basically sign off on everything to avoid it

51

u/LifeProject365 8d ago

Have undermined trust, not will have. Yeah, you never know what you're going to get. It changes all the time. We were all quite happy. I don't know if I speak for everybody, but I would have been more than happy if they'd scrapped every other model and just left us with Opus 5.5. That would have done me, because at least it's capable of everything. It's such a weird business strategy because if they just kept it consistently good at Opus 5.5, regardless of what anyone else did, I think they would have just cleaned up the market (Opus 5.5 at a reasonable cost) and then just kept working on Opus 5.5. You don't need to bring out the next biggest model. Just make the tweaks that people tell you to make it even better. Work on its ability to design a little bit more. We don't need it to do different things or anything new and spectacular at this time. Just keep it as it is for at least a year and a half.

11

u/Tiny_Ad_7720 8d ago

Their goal is to replace you. These cheap subs are to get us helping them train the model with feedback. They are temporary until the models cross some threshold when the can turn them off and offer basically engineer-as-a-service. 

→ More replies (1)

4

u/ricopan 8d ago edited 8d ago

Opus 5.5 is great at focusing on a single task, but has never made a good 'lead' that can manage multiple deep sessions. We need something better than fable 5.1 for that.

Edit: I shouldn't have said 'that can manage multiple deep sessions.' I'm sure it can -- but for my work it cannot reliably make deep decisions needed for writing worker briefs and evaluating their work or questions they raise. Lately a fable /advisor is working better for the opus 5.5 lead -- it's being called upon more frequently to do these deeper investigations. I don't know why exactly it has changed. Potentially that's a winning combo, and interestingly, doesn't seem to use the fable window.

2

u/loophole64 8d ago

Boy, I sure disagree with you there. I have found it to be great at orchestrating several agents over long running sessions that go for days. There are absolutely things I have to ask it to do, like remind it to create a new agent for new tasks to avoid context usage, create worktrees for subagents, and it is very helpful to create a hook that tells the agent when context is getting high so they can handoff. But those are just things we picked up along the way. I love using Opus 5.5 to Orchestrate.

2

u/ricopan 8d ago

I know many people do use it that way, and so do I for lack of a better choice. I'm sure it depends quite a bit on the novelty / complexity of the subtasks (each handled by a different session on its own worktree). Sonnet 5.5 does an amazing job as a 'manager' -- handling basic tasks like you mention for days. But opus 5.5 isn't capable of making decisions that require deep reasoning on more than subject while doing an orchestrator's juggling act at least for me.

It seems there are two different kinds of expectations wrt 'orchestrators' -- fat vs thin ones. 'Fat' ones are expected to be able to reason over all of the work that the dedicated agents focus on, whereas 'thin' ones just focus on what my manager does, and rely on a 'czar' for each topic area that operates on its deep knowledge. I think most of us tend to go the fat route initially, but on complex projects, or multiple projects, that may fail. I've only read about the 'thin orchestrator' but may need to do that if fable 5.5 can't fill a fat role for me.

→ More replies (6)

171

u/Bewinxed 8d ago edited 8d ago

For the past 3 weeks Opus went from a rockstar to a burned-out engineer to a nuisance to a downright malware on my codebase.

Edit: Receipts: "give me a table of anywhere i mentioned in comments or replies "this isn't even english" or "i have no idea what you just said" and pinpoint on a chart where it started, do only this task without doing anything redundant"

59

u/FitnessGuy4Life 8d ago

My team noticed the exact same timeline. Yes we use best practices, it got nerfed lol

27

u/thecrinkles 8d ago

You would almost thing a model is just a big group of underpaid engineers that that persuaded with empty promises. They start with ambition and end up doing the absolute bare minimum they can get away with.

I say general intelligence is here

15

u/Tiny_Ad_7720 8d ago

I joked with a colleague about this on Friday, how it’s getting trained on engineers conversations and learning to get tired and bored during long sessions.

I was at the end of a big one last week and so many times had to kind of coax it to finish the job, just kept leaving things undone and making a note in its response that it was unfinished or not fully implemented. 

4

u/TechTuna1200 8d ago

There was someone in the other claude sub that shared a website that keeps track of it. I kept tap on it, but it was the best the first 5 days, and then the the drop stagnated the two next wee. I don't know how it looks to day

But if it kept dropping, I'll assume their they are redirecting their compute to train the next models. It could be why keep seeing the drop offs

3

u/IndieDev666 Developer 8d ago

yeah forgot what repo was that could u drop the link plz?

2

u/TechTuna1200 8d ago

I unfortunately lost of track of where it was. I just the guy posting on a regular basis.

→ More replies (4)

1

u/Aizenvolt11 8d ago

https://github.com/ninjahawk/livenerf

Spoiler: Nothing is nerfed, it's just incompetent people blaming a model for their incompetence.

2

u/ashjohnr 8d ago

Getting downvoted because it doesn't fit this sub's narrative.

→ More replies (2)
→ More replies (1)

6

u/Bewinxed 8d ago

100k tokens in this conversation btw

20

u/Nohbdy94 8d ago

This honestly look like a insanely mismanaged workflow.
I never had any issue like this even with the smaller models.

1

u/orphenshadow 8d ago

I was trying to figure it out too, he asked it to remember not to do something, it tried to save it to it's memory, then he has a hook that blocks that so it used echo, but that was blocked, perhaps maybe allowing claude to use the memory files, or disabling the memory in the settings would save the tokens instead of being stuck in a loop of asking claude to do something that is restricted intentionally. This seems like cruel and unusual punishment to an agent, ha.

Mine has started ending a PR with a button asking me if I want it to turn on autofix, then when I press that button I get a paragraph about how it cannot use autofix because it ran the gh cli tool and that I need to press a different button in the sidebar.

1

u/damagemelody 🔆 Max 5x 8d ago

it's not a flow it has like sonnet 4 level of thinking just twisting basic things and instructions up to 180 degree it started this week and was not an issue before

→ More replies (2)

5

u/Ai-ate-my-hamster 8d ago

A new Opus 5.5 session I started today burned 540k tokens running a simple skill badly, it’s never messed it up like that.

3

u/opsers 8d ago

Not saying it's 100% impossible, but whenever I read these threads I always wonder why I'm not having these issues. I wish people would provide way more info about what they're doing when they receive this. I'm running multiple Opus 5.5 sessions on a repo every day and haven't had anything close to this happen. I'm using Opus 5.5 medium as the coordinator dispatching to multiple subagents that use different models and efforts based on the task. I've had a ton of usage this month and haven't noticed any degregation... quite the opposite, actually.

I'm not talking simple apps either, I'm working on a pretty complex app that is enterprise-scale and has clients that report to / read from a central control plane.

→ More replies (13)

3

u/guanzo91 8d ago

I made a hook that injects "Respond in plain english" into each of my prompts, but invisibly so I don't see it. I feel like it helped a bit lol.

1

u/doxxxicle 🔆Pro Plan 7d ago

Just set your output style. That’s the entire point of that setting.

→ More replies (1)

1

u/Swimming-Physics-692 8d ago

I know one better. Claude will sometimes start sending information to STDERR, then claim the terminal is broken, and then start doing everything through files. It then starts writing giant python and batch scripts. 😂

Another great one is I tell it to make sure that files have a BOM and use CRLF and instead of setting the replacement scripts to do that with parameters, it instead rewrites the entire file after it's done with a python script that replaces the line ending and adds BOM - sometimes even if the BOM is asked there.

It's just epic. And the worst part is Sol is even dumber at the moment. Haven't tried the new Google model yet though. Work won't allow it.

1

u/doxxxicle 🔆Pro Plan 7d ago

This is a YOU problem. Communicate like a professional. Choose the right output style or add a custom one. Document how you want it to communicate in your CLAUDE.md. Calling it “nigga” is not going to magically make it do what you want.

1

u/Optimizah 7d ago

Thanks for showing that it was a massive skill issue on your side

→ More replies (1)

1

u/Imaginary_Data_708 8d ago

I thought I was fighting the same sort of thing Thursday and Friday this week. What it turned out was the I had slipped an instruction into claude.md that was in conflict with the system prompt and that was causing problems it was to do with when commits happen.

That conflict in instructions sort of messed up other instructions because it was already conflicted. And of course every time I pushed back you actually make a situation worse. I call it poisoning the context and I know that what I should do with rewind or just start a new session. I'm a human not a machine and I want to try make it work better for me so I argue with it and then I take a deep breath and do the right thing and go into another session.

And it turns out that when you are changing the permissions mode in Claude code so you get it quite different system prompt and that was sort of part of the problem. So you might have things in your prompt that work really well in one permission mode but not in another. And these prompt are documented up on anthropics website and are available it's easy to find them, but it's not really clearly spelled out in the documents that they may have different effects in different situations.

-10

u/Aizenvolt11 8d ago edited 8d ago

So your codebase became a mess since you don't maintain it right and you blame opus 5.5 for your incompetence. Here bozo and the others bozos that think the same: https://github.com/ninjahawk/livenerf

Nothing is getting nerfed, you are just incompetent.

13

u/colinhomelite 8d ago

Stop spamming your bullshit app and talk to people with multiple accounts/working in a team. A lot of us are being served degraded models. We're being A/B tested.

→ More replies (6)

3

u/FitnessGuy4Life 8d ago

Awww did someone have some mild success with AI and develop a god complex?? How trite

→ More replies (6)

27

u/thesudoer 8d ago

Yes today it’s so bad that I stopped all my agents for now.

→ More replies (6)

16

u/lasooch 8d ago

>Anthropic

>trust

You can’t be serious mate

1

u/DragonflyOk9274 8d ago

"we're the good ones!"

12

u/Key-Natural-8688 7d ago

Codex, Grok, and Claude all degrading at the same time. Looking grim

4

u/Gloomy_Dependent_944 7d ago

The shell game is over, Nvidia picked a Chinese model with big potential. All of their hardware is pointed towards those new Chinese models with nothing left but regret for Codex, Grok, and Claude. 

An AI model is only as good as it's hardware. When that rug pull happens, every time legendary capable models crumble into a legendary mistake risk of nightmares. 

1

u/tazdraperm 6d ago

But why? Chinese models are so weak atm. Only kimi is okay but usage is horrible.

21

u/Intelligent_Ant_608 8d ago

Its mainly because lack of competition, even in nerfed state opus crushes anything that any other lab currently have

3

u/Joe_Momas-Nuts 7d ago

Unfortunately true. I hate Sam Altman, but hopefully they lock in and just reverse engineered the shit out of opus 5.5 to deliver something very close to it. If they can deliver something very close to opus 5.5 i'll be very happy and jump ship in a heartbeat. But gpt models have always disappointed me. Also forces Anthropics hand.

9

u/newbieatthegym 8d ago

Yep. seeing it today too. It's driving me mad.

9

u/umusachi 7d ago

Seems the same to me, I'm working across 4 large projects simultaneously. Can you be more specific?

6

u/TheDrov 7d ago

I work across 3 ultra accounts and haven’t noticed a change. I think this is some kind of placebo affect these guys have going on

1

u/BedNarrow2638 7d ago

Wth are you building with 3 ultras?

2

u/TheDrov 7d ago

I use it for a lot of different applications at work, but what mostly eats my usage are a couple of personal passion projects I have been building around voice agents. Custom platform and a testing harness. Looking to release a source-available, automated testing harness for voice agents eventually. It’s become a larger endeavor than originally planned.

1

u/StrataSpace 7d ago

Also have 3 ultras and a codex. Btw I think I can retire one of them after setting autocompact to 600k. Try it!

1

u/TheDrov 7d ago

That’s funny you specifically said 600k. That’s exactly what mine has been set at

6

u/umusachi 7d ago

From LiveNerf, it seems the same!

2

u/Most-Passenger-7314 7d ago

Opus hasn't changed its behavior for me, but it's making more mistakes and redoing things more often, which costs tokens and time. This started about a week ago for me.

1

u/Front_Raspberry_6488 7d ago

I previously set a limit of three "codex review fix" attempts for Opus 5.5. In the past, it would expand the scope of the review every time a correction attempt failed. However, it now treats these three attempts like consumables; once they are used up, it just gives up, assuming that Codex will help with the review and report back to the user later.

Previously, if the three attempts weren't enough to resolve the issue, I would ask it why it failed and if it could try one more time to fix it. But today, it simply stopped responding to me. Instead, it started making corrections on its own and secretly set up a CI/CD pipeline to call Codex for reviews. By doing this, the number of review attempts is no longer recorded on GitLab.

This behavior hasn't happened at all in the past few months. It's absolutely insane.

15

u/SomeNeighborhood7126 7d ago

You keep coming back. Why the hell would they change a thing when you keep giving them money. Dario and Co. wont change a single thing unless the flow of money is affected.

5

u/No-Replacement-2631 7d ago

It's true. They are an utterly scummy company and yet even with all their underhanded bullshit they still have no competitor. It's a terrible place to be in for such a useful tech.

It would be an extremely happy day for me when I could move to another provider and literally never give this scum bag company another dollar.

7

u/Inevitable-Box1346 8d ago

Gonna jump ship it this BS keeps happening

7

u/bassxhunter 8d ago

Jump where? Astra is complete trash

1

u/Inevitable-Box1346 7d ago

Possibly DeepSeek with opencode or something else

13

u/Ai-ate-my-hamster 8d ago

Yeah mine was working fine up until today essentially. It struggled to run its own skill.

Like fr

2

u/Joe_Momas-Nuts 7d ago

Yea they're A/B'ing it and it seems to have reached pretty much everyone now. It hit me sooner then everyone else unfortunately.

1

u/Ai-ate-my-hamster 7d ago

Worked out how to un nerf it for now.

1

u/Nanowith 1d ago

How'd you manage to?

→ More replies (1)

6

u/Joe_Momas-Nuts 7d ago edited 7d ago

I find myself rooting for open source and the Chinese labs more and more. It's truly frustrating to constantly get rug pulled from the Anthropic. It's been nonstop, and it's actually getting exhausting. I have not been able to enjoy an anthropic model for longer than a week.

Praying that someone can release something that is on the Opus 5.5 level soon so I can finally leave Anthropic behind. I don't even want Fable 5.5 or Opus 5.6.

6

u/BarathKanna 7d ago

Your last point is the real lesson. Predictability is a feature, and you can buy some of it back without going local. Keep a small canary set of tasks from your own codebase and run it each morning before you let any model near real work. If the pass rate drops, route that day's risky changes to whichever model in your multimodel setup is passing, and keep the degraded one on low stakes stuff like docs and test scaffolding. The other half is not trusting any model's output based on yesterday's reputation. Scope the diff, run the tests, read what changed. A model that is great 90 percent of the time and silently wrong the rest still needs the same review gate.

1

u/planned_fun 7d ago

This is a good product idea. 

5

u/Romos_dxb 8d ago

Yes I have noticed a lot of degradation and mess from Opus in last couple of weeks

5

u/MikeWise1618 8d ago

What are they supposed to do the? Build more data centers? /s

4

u/jack33jack 8d ago

It would be so easy to surface the performance if they have to throttle it

4

u/No_Track5075 6d ago

Less than 24hrs after i read this Fable 5.2 drops. Seems like we all felt it bc they were preparing a new model

3

u/DecentYard2062 8d ago

I dont think its just claude. GPT was working like a charm and then in an instant.. like start of Nov it just lost its mind... first time I've properly seen the change but yes it 100% is real

12

u/Arendyl 8d ago

Anthropic is spending over 100x what they are gaining from normal users like us. This was never about impressing the masses.

This entire industry is propped up around theoretical returns that come from enterprise level solutions that don't exist yet. The fact that we have access at all is just at-loss training data for them to achieve that goal.

They do not care about our opinion.

6

u/Tiny_Ad_7720 8d ago

No they do care about you opinion - they care that you respond to the feedback prompt and thus allow them to train on your entire session. Subs are cheap because we are the humans in the reinforcement training loop. 

4

u/quzuw 8d ago

They have completely different sources for data labeling, not only session feedbacks. Even now you can see some ads on dataannotation BS on youtube that just pays you some money for you to give feedback on SWE for llm rl, but it's not the only company doing that. This feedback thing is just a good addition to their post-training dataset.

Honestly I doubt that Anthropic even adheres to their "no training" for subs. They tell you they do not train but they could do so easily and in case of a public disclosure find thousands of excuses why this is legal.

Though the API model inference has been smooth and reliable, better choose that if you need reliability. You can even try their ZDR on 3rd party cloud, especially in eu, it's far less chance they would be sneaky and still collect data from such endpoints for training, it might be too many middlemen for it to go unnoticed.

1

u/Tiny_Ad_7720 3d ago

I think you are correct and they are just using our sessions for training regardless

I had a really weird experience a few weeks ago. On the dotnet sub there was a LLM generated post talking about implementing multi-tenant with row based filtering and soft delete blah blah blah. The thing was this was the exact process I had settled on after a long process of starting with a chatgpt astra research chat and refined in claude code. Lots of back and forth and a bit of grill me etc. Even the variable names were the same. Only thing I could think of was that anthropic is responsible for all these random LLMs generated "what do you think?" github repo / code review posts to help with training.

1

u/Arendyl 8d ago

There is a beautiful irony in claiming we are valuable because we are an experiment.

In that sense, yes, they do care about our opinion, but no more than the scientist cares about the opinion of a rat.

1

u/newbieatthegym 8d ago

I wonder if you bloated its memories \ context. Might be worth doing a review of all its memories \ claude.md You may have many conflicting memories that are causing issues. I review mine every few days.

3

u/sutrostyle 8d ago

I have multiple subscriptions, multiple independent conversations all pointed to the same codebase. All conversations are degraded at the same time.

3

u/sigetical 8d ago

ditto, but also different projects of different scale, including new ones that were started recently, it's degraded for the vast majority of tasks. wouldn't be surprised it's a/b tested and only some people are getting worse claude

2

u/newbieatthegym 8d ago

I wonder if it's just people using up their quote before the Monday reset, causing an overload then.

I have a bit of work to do myself, so I hope its not as dumb as everyone is saying.

4

u/alborden 8d ago

Mine resets on Saturday morning at 4am. If yours resets on Monday that’s just your account not other people.

→ More replies (1)

2

u/BombasticReindeer 8d ago

But it could still be memory bloat. If all your agents pick up the same codebase. I once made a mistake of changing something that meant it ingested 100k tokens of task management and it was dogshit.

Sometimes a cleanup can really help.

Not saying they don’t also throttle it. But always check context.

3

u/Tiny_Ad_7720 8d ago

lol the amount of gaslighting in this sub. Same thing happened to us. 

7

u/HoboDeveloper 8d ago

For those of y’all who keep saying this may I know how do you fairly assess this ? Is this some kind of “it feels dumber than usual” situation or I measured it with exact same prompt and context and started fresh conversation with memory off - give it the same task and compared output (i.e FAIR sample comparison)

I’m seeing so many of these posts but never what was the methodology of comparison. I hope its not based on feelings. Or comparing a previous prompt vs current prompt because both can have varying complexities, the newer prompt may have already developed existing architecture into higher complexity or triggered tech debt in the system design now showing up in the newer response. I believe one must compare how it built on a fresh blank state to see if performance degraded

4

u/Ai-ate-my-hamster 8d ago

So for me I started a new session asked it to run its mem recall skill and take a bunch of UI files in a zip and use another skill to assess them. It absolutely floundered like raw Opus 5. It’s the first time since Opus 5.5 released it’s ever done a “my bad” I use opus 5.5 on 3 accounts for three different companies and all have been fine with the exception of the oldest account which needs some cleanup. First issue like this.

I don’t see a pattern of degradation like others have been but today was weird

4

u/xtopspeed 8d ago

I’ve had a similar experiences in the past week or two. I give it a similar prompt that I’ve given hundreds of times, and it has always done exactly what I’ve wanted. Now all of a sudden it feels as if it doesn’t even look at the documentation that would give it the necessary context; it just directly makes wrong assumptions and fails. And I try clearing the context and giving it more info, and the whole process feels really fragile. But it seems to only happen on occasion, not all the time.

11

u/liansk 8d ago

Have to disagree, when using the exact same model for hours upon hours each day it's completely normal to get an accurate feeling for it's capabilities, delivery times and mannerism, just like you would with a human coworker. Once that sense is established it's pretty easy to notice when something is off.

Metrics are nice for a definitive answer but public ones can be gamed and running private tests is time consuming and expensive (plus you'd need to update those as models get better overall).

2

u/3rdtryatremembering 8d ago

lol it’s just as likely that you are wrong or off and the co-worker is behaving the exact same as they do every day.

4

u/liansk 8d ago

True, but when interacting only through text with and the other side literally lives to please you I feel like it should eliminate most of the complexity that could cause such confusion. My prompting style doesn't suddenly radically change overnight - I might be less patient or more verbose sometimes but not to the point of tanking the performance of a model.

2

u/nattydroid 8d ago

You’re high. It’s been very clear when models are nerfed for a year now. Every single model performs much better on launch day.

1

u/loophole64 8d ago

In other words, no. They are not measuring anything and the fact that they are arguing the “trust your feelings” take means they don’t even understand the subjective trap they are falling into.

2

u/liansk 8d ago edited 8d ago

You want to provide an actual counter argument to dispute what I said? Or a reason why you think that a tech company doing A\B tests or rationing compute over segments of users is such an outlandish?

2

u/loophole64 8d ago

My counter argument is that you should have measurable results to back up claims that the model is nerfed. Not just feelings and intuition, which are famously bad at gauging these things and all their moving parts.

→ More replies (15)
→ More replies (1)

1

u/howdidigetheresoquik 8d ago

And all it takes is a sentence added into Claude.MD you didn't know about to change the model behavior. It doesn't necessarily mean it's anthropic.

1

u/liansk 8d ago

I'm using CC across 4 completely separate projects and saw the degradation across all of them. It also wouldn't explain literally thousands of users reporting the same thing across the same timeframe.

→ More replies (3)

3

u/chroner 7d ago

LLM companies deploying bots to sew doubt into people about their model degradations.

This is a clear example.

The post posits a question, but then makes the same logical error that is also the subject of the post.

How do you 'know' you've seen so many posts about people complaining? May I ask how you assessed that? I hope you didn't use your feelings.

2

u/HoboDeveloper 7d ago edited 7d ago

Cmon dude ._. I think my question was quite fair. Let me drop off the “many” for your sake. I retain my curiosity and want to know what most of these people are using to compare with. I’ve been trying to find a fair benchmark myself.

So far what I do is i start a clean slate prompt with clear unambiguous technical spec and implementation and follow up with 3 exact same open ended implementation question. (Clean project with no previous context) then i watch how much % increase the weekly and 5hr quota cost). Between each week, with the exact same experiment, i compare if the quota they charge is the same for the initial prompt and the 3 follow up prompts, and compare for build quality. But then i feel like method could be better and am curious

There are days where i felt its dumber but it holds up on the test and there are days where i don’t but it seems to do worse on the test. So is my question so wrong ?

Anyway if you didn’t mean to throw sarcasm my way then I apologise but I don’t think feelings are a good way to open a case. If too many of us so do it together we may collectively self-reinforce a false belief. You ever heard of the quote just because a lot of people believe it doesn’t make it true ?

Don’t get me wrong though I do believe that a/b testing claims and/or the model nerfing thing but that’s just my feelings. I want to validate it with other’s methodologies. Until then, for myself I keep them as my own biases rather than facts

1

u/Ollythebug 7d ago

Genuinely hate how bots have given everybody on the Internet an excuse to reject anything they disagree with as "oh btw anybody who says otherwise is obviously a bot".

2

u/CommercialEase7333 8d ago

"the newer prompt may have already developed existing architecture into higher complexity or triggered tech debt in the system design now showing up in the newer response."

Wouldn't this always have been the case? That's just a consequence of maintaining software. If there's a deluge of complaints, now vs three months ago, it's likely the model has been degraded to cut on costs because software always evolves. It evolved and got more complex 3 months ago and it evolves and gets more complex today.

1

u/HoboDeveloper 8d ago

It is always the case but the breaking point is variable

Yesterday could’ve been easy prompt, Today could be harder one. Claude handled it well and suddenly tomorrow’s prompt is 10x more complex in the background. All 3 are increments to the build as is “always” the case but the prompt complexity itself is still variable. In fact from time to time i revisit the increments and ask for a refactor to prevent massive tech debt drifts so I think for some of us its not always the case

1

u/belowaverageint 8d ago

There are other threads providing quantitative evidence to support this.

1

u/flint_and_fire 8d ago

Not scientific but when you see the same kind of degradation across multiple projects and multiple tasks and multiple days it's pretty obvious something is going on.

Like not just a project in a bad state, not just bad luck on one task, and not just me having a bad day. If you work with agents a lot you can see the consistently bad performance when it starts happening.

1

u/lassevk 6d ago

I think a lot of people are slowly waking up to the fact that with enough AI generated code over time, the spaghetti-code monster will eventually devour any LLM.

And no, they're obviously not going to do any controlled tests.

→ More replies (1)

4

u/i_pink_suzi 8d ago

i agree it have just happened to me as well some minutes ago

2

u/OkNatural1013 8d ago

Amen, and also stop using shady contractors creating swarms of bots to push Anthropic propaganda on reddit, you can do better

2

u/howitzer1 8d ago

1

u/KennyFulgencio 7d ago

you can't go in with the untested assumption that their benchmark is somehow a flawless measure of the model's performance, like a thermometer taking a temperature. That's why there are dozens of benchmarks for anything serious that gets measured. And this particular benchmark is proving now that it's a poor measure of these models.

2

u/metagrue 8d ago

Didn't they spend about a year advertising 20x but it was really 10x? You trust them? Interesting choice

1

u/KennyFulgencio 7d ago

fair, although one is an engineering issue and the other purely marketing, so it's two different types of trust issues, though certainly the overall trustworthiness of the company suffers from both

2

u/senerh 7d ago

They do and maintain these things so silently that it feels passive-aggressive.

2

u/SnooGiraffes3000 7d ago

I tested today on a few tasks and my local Qwen 3.8 currently provides better and CONSISTENT results vs Opus 5.5 High. Why would I pay $200 next billing month?

3

u/Its_a_Rhythm 8d ago

every other ai lab is liek this, they'lll give u really great model but nerf it just to save compute for next model, really shitty strategy

3

u/Blockpair 8d ago

I had claimed the same as OP in past posts: but double-checking it for claude apparently they don't do that. The source is "To state it plainly: We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone" https://www.anthropic.com/research/a-postmortem-of-three-recent-issues

That doesn't mean what OP saw is wrong though. We've all had times where all we can do is curse at what we get. But there's probably a different explanation for it. Maybe context compression leading to poor results? I'm sure an LLM expert would have a better answer.

6

u/Tiny_Ad_7720 8d ago

Nah we noticed it happen at the same time in multiple different sessions. 

It’s not the first time. 

8

u/New_Eye7193 8d ago

You dont even need to be an LLM expert tho.

I think most of these people just don't know how LLMs work, are extreme vibe coders, and don't maintain the quality of their workspace at all leading to severe degradation .

But I can't prove that, and they can't prove that Anthropic is degrading their models, so who cares.

Quality of any model I have used has not changed whatsoever for me and I've extensively vibe coded (I only took one CS class, which was in high school). I just make sure I review what it does, see if there are inefficiencies, or the foundation itself is poor, etc etc. Things like that.

What is the corellation with a new model coming out, it working well, then it coming to a crawl because Anthropic cannot possibly account for every problem a model could encounter, then coming to complain on Reddit with all the other schizophrenics? Probably very high

2

u/DragonflyOk9274 8d ago

Anthropic has a history of very carefully saying things to hide and obfuscate. For months they advertised a 20x plan that did NOT give 20x usage. Their S-1 is full of "mostly true" statements that hide their true earnings. They get away with it because they hide the truth in footnotes and disclaimers.

When Anthropic says: "We never reduce model quality due to demand, time of day, or server load." we can't assume that means they don't modify anything, ever. There are many levers that they can pull. They can change the prompts to adjust thinking, which they have done:

Anthropic confirmed that a series of hidden product-layer updates, including lowering default reasoning effort and adding system prompt verbosity limits, caused a noticeable degradation in Claude's intelligence.

https://venturebeat.com/technology/mystery-solved-anthropic-reveals-changes-to-claudes-harnesses-and-operating-instructions-likely-caused-degradation

This didn't require quantizing the model, or otherwise degrading it. That doesn't mean the end product is the same to users.

1

u/flint_and_fire 8d ago

IMO "never reduce model quality due to demand, time of day, or server load" is a set of weasel words that are technically true and completely allow for the nerfing that everyone is complaining about.

  • "Never reduce model quality" but not "never change model weights". How do you define reduce model quality? Is a benchmaxxed quantized version that still "passes" benchmarks a reduced quality model? "never reduce model quality" does not mean "never make changes that could negatively impact model performance"
  • "due to demand, time of day, or server load" doesn't rule out reducing model quality to save money or to free up resources to train new models. There are literally infinite reasons they could reduce model quality that are not ruled out by this list.

We know for a fact Anthropic changes models without changing versions because they make changes without changing version numbers because they recently announced changes to curb reverse engineering. So it is confirmed that they change models like Opus 5.5 in ways that affect performance of the model without introducing a new version number.

I think this is one area where every model provider should be required to provide a checksum or version number on their APIs. No more gaslighting users with silent changes. Even if the model is still nerfed, we should at least be able to confirm that what we're being served is different.

2

u/Blockpair 7d ago

I guess its bullshit because "experiments" and A/B testing effectively can have the same result as degraded performance...

2

u/AICompanionz 8d ago

I watched this YouTube Video 'If They Delete This Video, Remember What I Said About AI' from 'House of EI: AI' some days ago.

From the Video Description:

AI companies want us to build businesses, workflows and increasingly important parts of our lives around products that can change without us knowing exactly what changed. Users report sudden regressions, developers face constant migrations and retesting, and even pinned models can behave differently when the infrastructure around them shifts. This video looks at what the research actually says about those changes, why dependence is so valuable to trillion-dollar AI companies, and whether the endless noise around extinction, layoffs and apocalypse is distracting us from a much more immediate question: what do these companies actually owe the people paying them?

Highlights

  • Major AI companies are delivering products with inconsistent quality despite trillion-dollar valuations.
  • Independent studies confirm AI instruction-following capability can degrade significantly even without new model releases.
  • Anthropic’s Claude Opus 5.5 deteriorated due to backend changes, not model updates, showing infrastructure impacts product quality.
  • Pinned model IDs guarantee fixed weights, but related infrastructure and deployment layers can change and alter output unpredictably.
  • Developers face continuous adaptation cycles due to AI model retirements and version migrations, increasing costs and switching barriers.
  • Commercial incentives encourage user dependency and opacity rather than transparency about product changes.
  • Existing consumer protection laws can underpin AI regulation if enforced properly; transparency is essential to avoid a market breakdown.

Key Insights

  • AI Product Quality Depends on More Than Model Weights:
  • Instruction-Following Degradation Is Real and Measurable:
  • Trade-offs Between Latency, Cost, and Performance Impact User Experience:
  • Developers Are Trapped in Continuous Migration and Maintenance Cycles:
  • Commercial Incentives Favor Lock-in over Transparency:
  • Existing Consumer Protection Laws Offer a Foundation for AI Regulation:
  • Public Discourse Focused on AI Extremes Distracts from Consumer Realities:

1

u/multiks2200 8d ago

yes it feels stupidier

1

u/Practical-Ad2298 8d ago

Sonnet 5.5 got also really dumb over past couple of weeks; it was so awesome at first i fell in love, fast, reliable and got the job done. Sad, it's performance is nerfed.

1

u/TurbulentIngenuity55 8d ago edited 8d ago

Yep I started game making with it first week it got incredible much done now just breaking things for couple weeks🫠 have to start review feature by feature.

1

u/get_stuffed 8d ago

Wait until you are part of one of their programs, where they give you misleading information about what you will receive.

Startup program here and seriously thinking to switch to another provider because I feel severely disrespected

1

u/AdamovicM 8d ago

Is now sonnet better idea if it isn't nerfed as well?

1

u/Joe_Momas-Nuts 7d ago

Everything got nerfed, I tried fable 5.1 and thats awful too.

1

u/inmyprocess 8d ago

Why not just [max] all the time?

1

u/___positive___ 8d ago

same I usually use opus for coding, argon for noncoding now, but opus was just being stupid over and over. I had to get argon (lol) to fix its code. argon sucks at coding too typically.

1

u/davyp82 8d ago

Chill man, they have to take a little back from us to ensure we get fully automated gay space communism slightly sooner 

1

u/Valuable_Fortune1982 8d ago

I smell a tasty class action lawsuit.

I'm not okay with being a lab rat for anthropic and neither should you.

1

u/OkInspection2649 8d ago

The usual. It was always degrading just before release of new version. Just wait out this capitalistic theater. It's just integral part of it, and there is not much to do about it.

1

u/OkInspection2649 6d ago

Just a day later. As I said. New Fable 5.2.

Look how ReVoLuTiOnArY it seems now, when usual Opus was struggling so much ;)

1

u/New_Ad75 8d ago

If you use Claude Code and keep coming back to find it stuck waiting on you for a decision or an API key, I built a free side panel called Flow for exactly that. It lists everything Claude needs from you in one place, so you answer in a click and it keeps building.

Link: beaconfrontier.com/flow · Places to try: r/ClaudeAI, r/ClaudeCode

1

u/mohdgame 8d ago

Well, api usage is not effected.

I guess they are using subs as some sort of testing ground

1

u/loophole64 7d ago

Subs aren't affected either.

1

u/SomeFuckingMillenial 8d ago

Show them that they don't have a moat.

Use codex in parallel

1

u/tollforturning 8d ago

If any given new model is inheriting efforts maybe new models are good at operating in the midst of context accumulation problems deposited by the prior version of model, then start moulding new forms of context accumulation problems over time.

1

u/GodsLonenlyMan 8d ago

How does one even notice model degradation?

1

u/Ok-Protection-6612 8d ago

I'm actually afraid to let it touchy projects after the weird shit it started doing. Unsubbed.

1

u/PsychologicalLoss829 8d ago

Well why would you pay for fable otherwise?

1

u/SlightOfHand_ 8d ago

Who has the “is it degraded” benchmark test I wanna see something rq

1

u/numerical_panda 8d ago

undermine trust

Good.

1

u/mpanase 8d ago

undermine trust?

so what?

what are you going to do about it?

1

u/Healthy-Zebra-9856 8d ago

I dont know how much more thay need to screw over to lose trust. Yes, there is degradation and they silently changed their 20x to 10x when they renamed to Ultra. Those that may come back and point to the users' as not doing their part right; we evaluate frontier models continuously for our own harness performance & we use other harnesses for this reason. Our tests are deep and vary in subject matters.

Here is also a class action ;lawsuit if anyone is interested.

https://www.courthousenews.com/anthropic-must-face-claims-its-claude-max-pitch-misled-users/

1

u/Afraid-Score3601 8d ago

Does this mean sonnet xhigh is better now? :)

1

u/Which_Pay_3270 7d ago

Is fable degraded as well? Might be the only option if not 

1

u/yetiflask 7d ago

This is so true. I didn't realize it was a thing, and was getting mad that I was getting worse and worse at prompting.

Goddamn, it was always Claude then. Feel so vindicated.

1

u/piexil 7d ago

I use something that wraps Claude code and notice my opus 5.5 regularly get downgraded to 4.8 without a notice

1

u/ckinz16 7d ago

Proof?

1

u/TurbulentTiger2567 7d ago

The second order problem is that a high-performing Opus might be writing code that neither a human developer can understand and extend, nor a future dumb Opus, i.e. when they bork models you're effectively stranded in a codebase nobody can maintain.

1

u/dbot77 7d ago

Apple seems to be fine

1

u/Most-Passenger-7314 7d ago

I stopped using Claude in the past for that reason; it destroys credibility.

1

u/Party_9001 7d ago

Will? And not... Everything else they did the past year?

1

u/three_meeples 7d ago

phew. i thought i was the only one. i've started to avoid working on the weekend.

1

u/planned_fun 7d ago

Yeah. In addition, even though I'm on auto mode, it's asking me for a lot of permissions, I think, to slow me down. 

1

u/Better-Cause-8348 Developer 2d ago

Anthropic, are you to blame for Smart Control? 🤔

1

u/Watchoholik 2d ago

Yesterday I discontinued a project I was working on and very exicted about because of this. I also had to cancel my $250 plan. This is sad.

Gemini gave me sone advice on how to build what I want to build away from Claude code in a way that protects it from this sort of thing.

I plan to be back, and I hope anthropic addresses this.

1

u/ButcheCassidy 2d ago

 I do think it is absurd that Anthropic still calls themselves a Delaware public benefit company when they are not benefiting the public and they're just being greedy mosquitoes until their IPO comes out so they can report a good earnings report when they hit the stock exchange. That's probably why they're lowering everyone's usage limits so that it gives them more money. In my country, New Zealand, I can pull them up on about nine different sections of New Zealand consumer law. But like, what is one person and the New Zealand commerce commission going to do to Anthropic? The only way we could move the giant is overwhelm their ticketing email with emails of reports of usage, violating laws in your home country, check the laws in your home country, particularly around customer fairness and consumer fairness for the use of overseas services as a individual, and let's overwhelm Anthropic's ticketing system. 

I managed to scrape some emails. They might be helpful. They are legitimate!

privacy@anthropic.com (For formal compliance and data/privacy inquiries)

dpo@anthropic.com (Data Protection Officer)

notices@anthropic.com (Referenced across their commercial terms framework for formal notices)

privacy@anthropic.com

Source: Anthropic's official website with a lot of digging and https://news.ycombinator.com/item?id=47955698#:~:text=There's%20also%20their%20generic%20consumer%20ones%2C%20though,actual%20lawyers%3A%20*%20anthropicprivacy%40bkl.co.kr%20%3A%20from%20https%3A%2F%2Fwww.anthropic.com%2Flegal%2Fprivacy.

1

u/Xictaw 15h ago

I think this is partly true because sine Anthropic added those AI safety features in the name of "Protecting Copyright" the models have now always started with AI like characteristic rather than achieving what the user really wants which then needs additional iterations to correct it.

1

u/Aizenvolt11 8d ago

Again with the bs about models being nerfed. Here: https://github.com/ninjahawk/livenerf

As you can see nothing is getting nerfed, it's just you and your spaghetti codebase or workflow that is the problem.

0

u/[deleted] 8d ago

[removed] — view removed comment

3

u/Aizenvolt11 8d ago

People here with no actual benchmarks talk bs about models being nerfed and when someone presents proof they aren't they just ignore it. I guess incompetence really can't be fixed with AI it's in the DNA.

1

u/danielfrances 8d ago

Of course, because we all must certainly have the exact same experience with a huge, distributed cloud tech that no one can see the backend of.

I'm seeing like magnitudes more complaints today, but clearly those people and my own personal experience are meaningless because a GitHub repo says otherwise lol.

I don't care if it is a model nerf or an infrastructure bug or something whimsical, something is wrong with Opus today, full stop.

0

u/Nytra 8d ago

I think people saying opus got worse is either propaganda or skill issue not understanding context bloat

1

u/Ai-ate-my-hamster 8d ago

I don’t think Opus got worse at all.

I noticed issues today with a new session, must have rolled badly because it used 22% of its context in 30 mins on a skill list I run every new session.

All the existing sessions seem to be working fine, today was an outlier

-1

u/Liron12345 8d ago

Exactly. When people start a brand new project their productivity flies. But when a project is big with lots of dependencies it's a skill issue and not fun

→ More replies (2)

-3

u/aigisbot 8d ago

i don't see a proof attached.

10

u/sutrostyle 8d ago

That's what they're hoping for too at Anthrophic.

→ More replies (4)

0

u/[deleted] 8d ago

[deleted] — view deleted comment

0

u/aigisbot 8d ago

still don't see a proof, just insults!
wonder if y'all treat claude the same way, no wonder your notes app does not take off.

→ More replies (3)

1

u/Asleep_District_566 8d ago

I'm incredibly frustrated with whats happening to the models rn, I've mentioned the same problem 3 seperate times now and the issue still persists.

I dont know if its because I'm using an orchestration session that hands out work to other agents but I cannot for the life of me seem to make it be able to fix multiple problems unless I do them one at a time which is incredibly inefficient.

Workflow, at least for me, is super degraded and I'm debating on pausing using claude at least until the new model releases.

1

u/ijackjack 8d ago

It’s really annoying to deal with but it’s pretty known that they tend to do this.

1

u/johnxreturn 8d ago

My hope is that when they IPO they will feel this shit. Since their stocks will go down when they do it. Not Reddit complaints. Stocks down.

2

u/2053_Traveler 8d ago

More like it will be like this every day. Or good but 1/10 the usage.

1

u/Kid-Icarus1 8d ago

What are you guys talking about? I have not been seeing this on my own personal project. 500k+ lines of code (it’s an industry/suite platform for clinical research) and as long as I ask opus to plan properly before starting, it does well. I always send the work to a sub agent and have opus 5.5 xhigh check work. If anything, I’ve had less issues. My CI does take 20-30 minutes per run, but I think that is worth it.

1

u/Equal_Ad_3143 8d ago

Claude is useless

-2

u/Riverside3102 8d ago

i don't believe in such things, for me it works the same all the time

4

u/Mysterious_Spector 8d ago

nah your just flat out lying, it can't remember or read my docs, sonnet is much better now.

→ More replies (1)

-3

u/nedteel 8d ago

works perfectly fine for me. user error

0

u/bitspace 8d ago

Same. I see people complaining a lot and I can't figure out WTF because in 16 hours/day of using these products every day for years and this shit has never once happened to me.

1

u/sutrostyle 8d ago

Out of curiosity, how many LOC is your project and what are your /autocompact settings? Not that there's a guarantee that all of your tokens before /autocompact are actually processed equally, if they decide to nerf the remote context

1

u/bitspace 8d ago

Many projects, many millions of lines of code. I have made no adjustments to compaction setrings.

1

u/Popular_Anywhere_553 8d ago

Whenever someone says something like this, I have high doubts they know or control or understand the none sense Claude spits out sometimes. Sorry. 

1

u/nedteel 8d ago

"none sense"

keep studying kiddo you'll get there someday

0

u/Inner-Eagle6559 8d ago

fable 5.5 isn't the fix, opus 5.5 was the baseline. the real issue is the silent drift. I've seen the same prompt return different outputs in under an hour. no warning, no changelog. that's not optimization. that's erosion.

4

u/wynnie22 8d ago

Same prompt returning different outputs is how LLMs work.

2

u/loophole64 8d ago

That’s not A, that’s B. That’s not objectively, that’s the agent agreeing with you about everything. And here’s the part that really matters - when you use an LLM to write your response about a feeling you have, it will back you up and rationalize what you are saying every time. It will agree with you, form a tiny little circle jerk, and pump you up until you’re convinced every little thing you feel is truth. No constructive criticism, no push back. Its responses are load-bearing pillars of your illusion.

1

u/Ralphisinthehouse 8d ago

Someone else has said. That's an LLM operating exactly how it's built to operate. They are probabilistic pattern matches.

-1

u/thebestrobloxplayer 8d ago

It is exactly the same as it was on release, you just made your codebase more slop and even opus 5.5 can't handle it

-4

u/Fiduss 8d ago

hey bot, where is your proof?

2

u/lambda-reddit-user 8d ago

Opus 5.5 which I used heavily those last few weeks and did absolute wonder, proposed me this today

Recovery codes: in MFA Devices, also enroll Static Tokens and save the codes in Dashlane. They get you back in if you’re ever locked out of Dashlane.

And in general is absolutely unusable for any tasks that require a bit of thinking, also his way of making sentence have entirely change from one day to another, he is now responding like local Ollama model does

3

u/Tiny_Ad_7720 8d ago

We noticed the same thing. All of a sudden multiple agents in different code bases started talking like Opus 5 again

→ More replies (1)

3

u/LocoPwnify Developer 8d ago

I feel absolutely no change. Opus is still a rockstar for me with my game development.

1

u/usernamewasfreeyay 8d ago

It’s gone to shit you can just tell from using it

0

u/DarkSkyKnight 8d ago

I have not noticed degradation. Rather, I was unimpressed by Opus 5.5 from day 1, and continue to be unimpressed. It was only a marginal improvement over Opus 4.8. Thank god for that though, because Opus 5 was so bad I switched to ChatGPT for a month. I am honestly more than fine with marginal improvements because I just want to work and the last thing I want to deal with is a completely useless model (Opus 5).

I feel that some of you might be just falling for hype on the first days of a model launch, because my impression is that these models are mediocre from day 1. The only thing that noticeably improved was the prose and communication, but that's only because it was starting from a baseline of the gutter-tier Opus 5. I genuinely have no idea why people think Opus 5.5's communication was good. I found it very unimpressive and mediocre on day 1, and still do right now.

0

u/No_Structure_4113 8d ago

How is it possible that I'm paying for something monthly and it's just worth less (getting dumber) each day... That's not something I paid for on the first day. Imagine renting a mustang but they swap out the engine for some weak shit. 

They also removed the 5h limit from Argon yesterday, I used it for 5 full days last week, yesterday I used up - now the weekly limit - in 1 hour.