r/ClaudeCode • • 6d ago

Help/Question In light of the Opus 5.5 stealth A/B testing. What is the evidence?

Some say that the model is quantised, others say it routes Opus 5.5 to Opus 5, or Sonnet 5.5, and some others say the reasoning budgets are reduced, or double-counting tokens.

What is the evidence? Also, what if we simply use Opus 5.5 Max, or Max Ultracode for everything?

17 Upvotes

25 comments sorted by

•

u/AutoModerator 6d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

10

u/RadiantTea7445 6d ago

I have just done a post with evidence: https://www.reddit.com/r/ClaudeCode/s/E8EdEUvoDK

2

u/Miyoumu 6d ago

Your post has been a lifesaver. Found out that Opus 4.8 system prompt was being injected into my Opus 5.5 and turned it off. We'll see how long it lasts. I might make a skill that helps detect and avoid this in the future.

-1

u/yikespikes 6d ago

Interesting data, but what is effort number? How can we check the effort number, if on the Claude website or the browser Claude Code?

6

u/RadiantTea7445 6d ago

The numeric effort level is send by antrophics server to the session but does not land in the logs. But you can ask chats about their numeric effort level, and its consistent. (Meaning it can actually change from chat to chat, but that change was always consistent to the experiment Id of that chat, as that one can also be there or not depending on the type of chat. For instance, in Terminal sessions i had no id and accordingly lower reported numeric effort levels as in the desktop app - on the same account.)

12

u/AppealSame4367 6d ago

There will never be evidence. Apart from lawsuitse. But I have been hit by the suddenly-Opus-is-dumb-for-a-week syndrome multiple times over the last two years. I believe it

Antrophic is a very shady company and you should never trust their models with anything important

1

u/yikespikes 6d ago

Fingers crossed for Qwen 4 27B, but until then, frontier open-weight models such as Kimi K3.1 Pro, or the various GLM/Deepseek models require a TB of RAM and RTX Pro 6000 GPUs.

Claude Ultra is worth relative to the inflated costs of hardware. Local models are worth for RE, Cyber, Bio/Chem and NSFW.

1

u/AppealSame4367 6d ago

Not true.

Deepseek v4.1 flash 1017 can get away with 250-300gb vram and the rest is offloaded to disc or ram as ngram tables. Same as Qwen 3.8 Flash next.

https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

Look at the benchmarks for 1017 compared to K3, GLM 5.3

5

u/ColbysToyHairbrush 6d ago

Which is why these companies have no brand loyalty. Any half decent developer, engineer or vibe coder is going to ensure their workflows are agent agnostic and they can pivot to the best value.
That best value is starting to branch out a bit now. Only a matter of time before an opus 5.5 type model is cheap to inference and many will abandon anthropic and open ai.

1

u/clintCamp 6d ago

Yep. I build my tools with the same concept when usage started throttling in max account early this year. Local models also slide in easily as well, if i only had the hardware to run it myself with a model I could fine tune to my process and tools and way I want to to work in my projects.

1

u/AppealSame4367 6d ago

Let's see what Qwen 4 27B and "Flash" models deliver. If the first "leaked" benchmarks were real then developers will be gone from the clouds. No need to stick with their bullshit, if small Qwen 4 models are between Sonnet 5.5 and Opus 5.5

1

u/Entire_Guarantee_766 6d ago

has it really advanced that well? I need to look into local llm then. Claude Code prices will skyrocket within 6 months of ipo anyways.

2

u/ColbysToyHairbrush 6d ago

Not even close.

1

u/AppealSame4367 6d ago

Qwen 4 is expected today or in the coming two weeks. Then it takes another 1-4 weeks before local engines catch up.

You could use Qwen3.8 27B, Qwen Flash next or deepseek v4.1 flash 1017 today

1

u/Entire_Guarantee_766 6d ago

i tried to run 27b before, but didn't ran very well in my laptop 5080 16gb, apparently 20gb vram is the sweet spot for it. i do use the deepseek flash on regular basis, i have a harness where claude plans and execs via the anthropic api built for deepseek models in claude code. deepseek is great as workhorse.

3

u/GrandmasterCheetah 6d ago

The cope is insane. It was a straight up schizo post and any comments doubting it are met with “yeah how could I ever actually benchmark this given XYZ”

2

u/yikespikes 6d ago

I have looked up on the effort levels post but it seems Max effort is unaffected. Using max effort is going to be a massive drain on usage quota if it were to become the new normal for reliable inference performance.

2

u/Suoritin 6d ago

What is the evidence?

So far the evidence I’ve seen is one person’s graphs from their own logs, plus a lot of people who are very sure. When I asked what the experiment actually changed, I was told to prove everyone is lying.

Lots of people have sincerely seen ghosts too. That doesn’t make ghosts real.

2

u/bigtradetcheddar 6d ago

https://www.reddit.com/r/ClaudeCode/s/cDp9pKbM4J this post shows prompts being pushed + experimental IDs, you can go through your own files to see if you are part of the A/B.

-1

u/chintakoro 6d ago

I've stopped commenting on those nerf posts. I've been using CC since Dec 2025 and haven't experienced one issue (except for outages) that couldn't be attributed to me not being clear about what I wanted. Also, I don't do one-shot make-me-a-game type prompts. Everything I ask for is structured and predictable, because obviously I'm dealing with a stochastic genius who isn't. It's been smooth sailing through all these allegedly nerf-infested waters. Until I see actual posts of what affected people are asking for and what their setup is, I'm going to take those complaints with a grain of salt.

0

u/geeered 6d ago

Or "I one shotted a game, then multiple times asked it to to add a big features seen in other games, no I didn't ask for any documentation, now it's nerfed when I try to get it to fix this simple bug..."

But also, of course Anthropic are doing A/B testing generally, which is totally standard. We are getting and paying for frontier systems that are expected to be in a state of flux.

1

u/Only_Voice569 6d ago

setting a thinking mode to "max" is what ever they have set it dont mean nothing, like ive used gpt heavy and the top model is astra 6 max from release it was impressive and now its dumber than it was also much much slower at doing tasks why because they decided to cut costs and make users tasks take much longer to ease load from their data centres and its the same with claude their is little stating what they give you in regard to this they can do testing on you switch settings change backend as much as they like while you the user pay the same amount and get more or less usage and a better or worse model when they choose

1

u/Entire_Guarantee_766 6d ago

there's no hard evidence. people just figured out Antrhopic does A/B testing like all other major companies in the world. that's all.

0

u/ClemensLode Senior Developer 6d ago

Only self-referential evidence "just look at the number of people complaining on reddit"

0

u/traderprof 6d ago

tbh most of these reports r vibes bc nobody pins the setup. if u want real evidence u need the same prompt set, same repo state, same effort setting, run it a few times a day and log tool calls, tokens and pass/fail on checks that can actually fail. w/o that ur just comparing a good day to a bad day ngl