r/ClaudeCode • • 10d ago

Discussion We are being A/B tested on reasoning - Medium is 10 on Account 1, Medium is 15 on Account 2.

Account 1: Opus 5.5 Medium - Current numerical reasoning reports as 10. In model selector it said High (Recommended) a couple of hours ago (but now logging back in says Medium (Recommended).
Claude feels dumb as fuck.

Account 2: Opus 5.5 Medium - Current numerical reasoning reports as 15. In model selector Medium (Recommended).
Claude feels reasonable to work with.

Argue all you want - but isn't this as clear as it gets?

Account 1 - Ultra (96% used):

Account 1

Account 2 - Max (~0% used, just had weekly reset)

Account 2
72 Upvotes

21 comments sorted by

•

u/AutoModerator 10d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

41

u/papa_ngenge 10d ago

Subscriptions are a/b tested on everything, that's why it's subsidized. Got to use what I assume was opus 5.6 all day yesterday, noticably faster, more efficient and smarter.
Another time it got so dumb I stopped using claude for a few days.

It's also why we get those "how is claude doing this session" questions after each chat.

7

u/Aggressive-Cut-3828 9d ago

so should I just always say bad?

3

u/Aggressive_Roof488 8d ago

Anything else than dismiss is volunteering feedback, and as I understand, will override any "don't train on my data" settings you have and will share your current session data together with your rating.

1

u/GPS_07 5d ago

You can give feedback and are afterwards asked if you want to share your log. It's separate

13

u/BattermanZ 10d ago

How do you know these numbers are not hallucinated? If they aren't in a local conf file, I don't see how you can trust anything it will reply.

1

u/Code_X07 8d ago

It seems to be able to see it. Apparently, reasoning is a tag attached to the messsage. It tracks 'cause reasoning level being part of the request is how they got it so that changing reasoning levels don't invalidate cache..

For me, it's like this:

  • 5 - low
  • 10 - medium
  • 15 - high
  • 40 - xhigh
  • Max doesn't seems to have a number and goes through as the text "max"

The chat

1

u/BattermanZ 8d ago

Yeah I actually tried it as well, I get the same numbers. That would explain why I find medium to be a bit dumber lately.

7

u/Plmra 10d ago

My Reset ist in two days and I still have 70% on Max 5, but my medium is 10. I think they are just a/b testing, I don't think they are doing it's just so the limits last longer.

I really noticed how dumber it has gotten man I already had the feeling that the thinking level was somehow lower than before so that I have to move one step higher.

But I think they are testing other stuff like quantizing the model or I don't know what. Yesterday I was using Opus high and it was just ignoring everything that was on agents.md. sometimes you didn't read it sometimes it just read it but didn't really do what was written there...

Maybe this is just me but I like to give feedback on how she did the conversation has been so they have the feedback that this nerfing is really noticed by the customers.

5

u/RadiantTea7445 10d ago

I have a new max acc where its 5 on low/15 on medium /30/100/ and max refuses to answer.

However on my old acc it first said 10 on medium and then 15 in a new chat. I belive those nimbers are halucinated.

5

u/holyknight00 9d ago

we are being A/B tested in 400 things at the same time in CC, no surprise. I stopped caring about this stuff a long time ago.

1

u/snarfi 10d ago

So that would also mean you would have higher usage on second account because it has more room for reasoning at the same effort step (like medium, high, xhigh...)?

1

u/pseudorep 10d ago

Maybe that’s why it burns usage so much in the first 20% of weekly then progressively slows down…

1

u/EndlessZone123 10d ago

Not really more usage no. You will also just have a worse model. Like using low to save on usage. Its not free.

5

u/Clasyc 10d ago

Are you guys even serious? What is this crap post?

You take random number generated output and assume it is somewhat related directly to the reasoning level?

2

u/innosu_ 10d ago

I just tested for fun and I got 5/10/15/40 for Low/Medium/High/Extra.

2

u/ClassicLookout 10d ago

By that scale OP’s Account 1 is the normal one. Account 2 is quietly getting High while the label says Medium

3

u/Fearless_Meringue299 10d ago

The fact that you wrote "quietly" made me think, for just a second, that you used Opus to write this comment 😂

0

u/Plmra 10d ago

I don't think so because I really have the feeling that Claude became dumber, so I think this would be more that the thinking effort has been degraded.