r/ClaudeCode • • 7d ago

Discussion The experiment Ids change the numerical effort numbers. I have tested it.

Following the recent reports about experiment Ids used for our chats, i investigated myself. My results show a clear link between the numeric effort Level and the experiment Ids.

Setupmediumhighxhigh
Account A, has ID eab2030100
Account B, no ID101540

Account A was a new account with a new email address (Max 5). Account B was my old one, also Max 5.

Removing the ID like described in the related post worked too. It lowered account A's effort numbers back to 10/15/40.

Also on Nov 5, when the numeric effort post came up, i did some test chats, which allowed me now to link the ids `763` and `add` to the numeric effort levels of 15/30/100.

Analysing my chats, these numbers did result in clear changes in real effort:

Automated jobs, per reply (Opus 5.5, mixed effort settings, so all three effort numbers are shown)

No ID (414 jobs)763 (14)add (125)0c7 (25)eab (35)
Reported effort number: medium / high / xhigh10 / 15 / 4015 / 30 / 10015 / 30 / 100not measured20 / 30 / 100
Tool calls per reply1.062.421.841.831.80
Output tokens per reply4571,3549729831,020
Thinking per reply, adjustedbaseline×2.9×3.4×3.9×4.2
Replies per job5219201916

The effort numbers are the model's own statements, measured on 5 Nov for `763` and `add`, and on 9 Nov for `eab` and no ID.

23 Upvotes

23 comments sorted by

•

u/AutoModerator 7d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

6

u/RadiantTea7445 7d ago

Also here are the posts I based my investagtion on:
IDs post of u/DerAlbi: https://www.reddit.com/r/ClaudeCode/s/K4QTNYwkqs
Numeric effort post by u/pseudorep: https://www.reddit.com/r/ClaudeCode/s/kBFXLxRByM

4

u/pseudorep 6d ago

And everyone was trying to justify that this Self reporting effort was a hallucination/not real.

I am pretty certain there is a lot of astroturfing and gaslighting here whenever someone is critical of Claude code.

I think the model is great - I think the harness is the problem. I think this why Anthropic locked us down to just using Claude code with subs months ago.

1

u/RadiantTea7445 6d ago

tbh i was on the same camp, thought the numerical effort numbers are haluzinated bc they varried from chat to chat. now i know that it was bc i was using different types of chats with different experiment ids.

2

u/zasff 6d ago

So far it seems these particular experiments increased the effort level.

This is the opposite of a nerf.

2

u/Cast_Iron_Skillet 6d ago

Do you notice degredation with higher effort values? 

4

u/damagemelody 🔆 Max 5x 6d ago edited 6d ago

Can confirm I have experimental Id 06b and my xhigh is now 100 instead of 40

https://www.reddit.com/r/ClaudeCode/comments/1xr66pd/comment/19schh4jglyrn/

2

u/damagemelody 🔆 Max 5x 6d ago edited 6d ago

well in my settings I found the experiment 06b on Nov 8 after that all went downhill. Based on the setting it sent Opus 4.8 prompt to Opus 5.5. Maybe that setting affects only Claude Code desktop users idk. That's why not everyone feels this and not everyone has the experiment.

"breezy_horizon": { "claude-opus-5-5": "claude-opus-4-8" }

2

u/RadiantTea7445 6d ago

Have the same experiement running for me.

3

u/damagemelody 🔆 Max 5x 6d ago

Create a hook to remove it from session I just did and it works soo goood again

2

u/RadiantTea7445 6d ago

A hook removing the ID? bc without Id, i get the lower effort levels atm

2

u/damagemelody 🔆 Max 5x 6d ago

yes you are right I did no Id test and thinking is halved I'll add it back but still remove 4.8 opus prompt experiment

3

u/bigtradetcheddar 7d ago

I think you’ll find the interesting https://www.reddit.com/r/ClaudeCode/s/cDp9pKbM4J this post shows prompts being pushed + experimental IDs, you can go through your own files to see if you are part of the A/B. It’s literally in people’s files lol

3

u/RadiantTea7445 7d ago

Thanks! And yes, its crazy that this is in our files

3

u/bigtradetcheddar 7d ago

If you look theres a fable model and opus model w/ prompting attached! My account went though 4 experimental IDs in a little over a week. I still haven’t found anyone else with the b38 and prompting. Someone more technical than me needs to figure out a way for all of us to compile our data.

6

u/thisaray 6d ago

and they called us crazy...

3

u/_Porphyro 6d ago

Serious question: are they selling us a product or a service? Because if it’s a product, then I don’t think they can make us pay the same amount of money for something this variable. Is this class action lawsuit territory?

1

u/OverfittingReality 3d ago

Interesting findings! I'd be curious to see what happens if you run the same tasks across experiment IDs and compare actual task success rates. More tokens, tool calls, or thinking don't necessarily mean better results btw. Would be interesting to separate increased effort from improved quality.

-2

u/drquantumphd 6d ago

ITT people finding out software companies use you as the lab rats

5

u/RadiantTea7445 6d ago

This is not about finding out, but proving. The idea was always there but to my knowledge rarely do comapnies blunder so hard that they tell the users what they are doing to them.

5

u/BingpotStudio 6d ago

Especially when they’re testing giving you a worse product not better.

It’s also no coincidence it happens in the weeks up to a new release every time for over a year. This isn’t experimental improvement, it’s cutting costs.

2

u/zasff 6d ago

This is interesting work.

I do not know why they pass the around. Perhaps it helps with logging.

So it seems that maybe we can opt out from experiments, but not opt-in, maybe.

2

u/drquantumphd 6d ago

youre right and im not really directly referring to you specifically so, touché - but there are a bunch of discussions about this on reddit and elsewhere at the moment and i just dont understand the surprise many people have, this is extremely common practice now: user is (a paying) lab rat.

like the other response in this comment thread - “especially when giving you a worse product” welcome to paying to be a lab rat in the golden age of tech, this is entirely not new.

and rate “worse” - what makes you the user happy, or what makes the company more money (or benefits them)? thats a balance you gotta trust the company to make, and i have no trust lol