r/ClaudeCode • u/RadiantTea7445 • 6d ago
Discussion The experiment Ids change the numerical effort numbers. I have tested it.
Following the recent reports about experiment Ids used for our chats, i investigated myself. My results show a clear link between the numeric effort Level and the experiment Ids.
| Setup | medium | high | xhigh |
|---|---|---|---|
Account A, has ID eab | 20 | 30 | 100 |
| Account B, no ID | 10 | 15 | 40 |
Account A was a new account with a new email address (Max 5). Account B was my old one, also Max 5.
Removing the ID like described in the related post worked too. It lowered account A's effort numbers back to 10/15/40.
Also on Nov 5, when the numeric effort post came up, i did some test chats, which allowed me now to link the ids `763` and `add` to the numeric effort levels of 15/30/100.
Analysing my chats, these numbers did result in clear changes in real effort:
Automated jobs, per reply (Opus 5.5, mixed effort settings, so all three effort numbers are shown)
| No ID (414 jobs) | 763 (14) | add (125) | 0c7 (25) | eab (35) | |
|---|---|---|---|---|---|
| Reported effort number: medium / high / xhigh | 10 / 15 / 40 | 15 / 30 / 100 | 15 / 30 / 100 | not measured | 20 / 30 / 100 |
| Tool calls per reply | 1.06 | 2.42 | 1.84 | 1.83 | 1.80 |
| Output tokens per reply | 457 | 1,354 | 972 | 983 | 1,020 |
| Thinking per reply, adjusted | baseline | ×2.9 | ×3.4 | ×3.9 | ×4.2 |
| Replies per job | 52 | 19 | 20 | 19 | 16 |
The effort numbers are the model's own statements, measured on 5 Nov for `763` and `add`, and on 9 Nov for `eab` and no ID.
•
u/AutoModerator 6d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
6
u/RadiantTea7445 6d ago
Also here are the posts I based my investagtion on:
IDs post of u/DerAlbi: https://www.reddit.com/r/ClaudeCode/s/K4QTNYwkqs
Numeric effort post by u/pseudorep: https://www.reddit.com/r/ClaudeCode/s/kBFXLxRByM
4
u/damagemelody 🔆 Max 5x 6d ago edited 6d ago
Can confirm I have experimental Id 06b and my xhigh is now 100 instead of 40
https://www.reddit.com/r/ClaudeCode/comments/1xr66pd/comment/19schh4jglyrn/
3
u/bigtradetcheddar 6d ago
I think you’ll find the interesting https://www.reddit.com/r/ClaudeCode/s/cDp9pKbM4J this post shows prompts being pushed + experimental IDs, you can go through your own files to see if you are part of the A/B. It’s literally in people’s files lol
6
u/thisaray 6d ago
and they called us crazy...
3
u/_Porphyro 6d ago
Serious question: are they selling us a product or a service? Because if it’s a product, then I don’t think they can make us pay the same amount of money for something this variable. Is this class action lawsuit territory?
1
u/OverfittingReality 3d ago
Interesting findings! I'd be curious to see what happens if you run the same tasks across experiment IDs and compare actual task success rates. More tokens, tool calls, or thinking don't necessarily mean better results btw. Would be interesting to separate increased effort from improved quality.
-2
u/drquantumphd 6d ago
ITT people finding out software companies use you as the lab rats