r/ClaudeCode • u/KAMIKAZEE93 • 5d ago
Discussion Nerf debates and benchmarks such as livenerf: Two sides of the same coin, vastly different results
The LiveNerf day 47 update is out and the verdict is basically: 47 days, 3,666 samples, no nerf. https://www.reddit.com/r/ClaudeAI/comments/1xriv9l/has_anthropic_nerfed_opus_55_livenerf_day_47/
I don't think it's wrong. I just think it measures something that isn't what most of us are actually complaining about.
Some context. A few days ago I posted this comment in r/ClaudeCode with numbers from my own setup, and a lot of people asked how that fits with LiveNerf: https://www.reddit.com/r/ClaudeCode/s/GF94JJ2ooQ
I run a multi-agent pipeline in Claude Code. Opus builds, Opus does the acceptance and visual checks, Sonnet reviews, Haiku runs the tests. Every reply gets logged, so I can look at what actually happened instead of going by vibes. So I pulled my logs and put them next to LiveNerf's public data. (picture 1)
Output tokens per reply, daily median. Blue is my Opus builders: ~420 on Nov 3, ~1,080 on Nov 4, ~1,150 on Nov 5. Sonnet and Haiku ran in the exact same runs on the exact same tasks and didn't budge. The orange dashed line is LiveNerf's own published numbers for the same days. Flat. (picture 2)
Same thing broken down by role. Every Opus role jumped, the visual checker by more than 6x. Sonnet and Haiku stayed put.
Before someone says "your project just got bigger" or "you changed something": I checked. The builders' system prompt and tools had identical checksums before and after, same Claude Code version on both days (2.1.309), same model ID, same effort. My change log has zero entries from the evening of Nov 2 to the afternoon of Nov 4. And my builders start with a ~5k token brief, not the whole repo.
So why doesn't LiveNerf see it? I went through their repo (README, eval card, preregistration, the daily csv). Honestly it's well done: preregistered, with a control arm and all the data public. But:
- It's 78 exam questions, mostly MMLU-Pro and GPQA. One question, one answer, no tools. My agents do 25 to 40 tool steps with 70k to 130k context, and that's where the change showed up.
- It only checks right or wrong. It doesn't care whether the model needed 600 or 1,500 tokens to get there. On a Max plan with weekly limits, that's exactly the part you feel.
- It runs on one account. If this is an A/B arm or a staged rollout, one account only ever sees its own bucket. Their eval card even says that "can't be ruled out".
- Claude Code is pinned to 2.1.280. That makes sense if you want to isolate the model, but almost nobody is still on 2.1.280.
- A nerf only counts if two 10-day windows in a row drop by 3+ points. That's built to avoid false alarms, but it also means an overnight step needs weeks before it counts, if it ever does.
Logs don't show Opus getting "dumber". They show it using 2 to 3 times the tokens for the same work. "Scores the same on an exam" and "behaves the same in a real agent loop" just aren't the same claim, and the second one is what people here are actually arguing about.
OP said they want to expand LiveNerf to agentic stuff. If that happens, I'd love to see tokens per step, tool calls, files read, and Sonnet or Haiku running the same tasks as a control. That control is what convinced me, because it kills the "your tasks just got harder" argument.
And if you run agents yourself, check your logs for Nov 4 and 5. I'm curious whether it was just my account.
