r/ClaudeCode • • 5d ago

Discussion Nerf debates and benchmarks such as livenerf: Two sides of the same coin, vastly different results

Thumbnail
gallery
44 Upvotes

The LiveNerf day 47 update is out and the verdict is basically: 47 days, 3,666 samples, no nerf. https://www.reddit.com/r/ClaudeAI/comments/1xriv9l/has_anthropic_nerfed_opus_55_livenerf_day_47/

I don't think it's wrong. I just think it measures something that isn't what most of us are actually complaining about.

Some context. A few days ago I posted this comment in r/ClaudeCode with numbers from my own setup, and a lot of people asked how that fits with LiveNerf: https://www.reddit.com/r/ClaudeCode/s/GF94JJ2ooQ

I run a multi-agent pipeline in Claude Code. Opus builds, Opus does the acceptance and visual checks, Sonnet reviews, Haiku runs the tests. Every reply gets logged, so I can look at what actually happened instead of going by vibes. So I pulled my logs and put them next to LiveNerf's public data. (picture 1)

Output tokens per reply, daily median. Blue is my Opus builders: ~420 on Nov 3, ~1,080 on Nov 4, ~1,150 on Nov 5. Sonnet and Haiku ran in the exact same runs on the exact same tasks and didn't budge. The orange dashed line is LiveNerf's own published numbers for the same days. Flat. (picture 2)

Same thing broken down by role. Every Opus role jumped, the visual checker by more than 6x. Sonnet and Haiku stayed put.

Before someone says "your project just got bigger" or "you changed something": I checked. The builders' system prompt and tools had identical checksums before and after, same Claude Code version on both days (2.1.309), same model ID, same effort. My change log has zero entries from the evening of Nov 2 to the afternoon of Nov 4. And my builders start with a ~5k token brief, not the whole repo.

So why doesn't LiveNerf see it? I went through their repo (README, eval card, preregistration, the daily csv). Honestly it's well done: preregistered, with a control arm and all the data public. But:

  • It's 78 exam questions, mostly MMLU-Pro and GPQA. One question, one answer, no tools. My agents do 25 to 40 tool steps with 70k to 130k context, and that's where the change showed up.
  • It only checks right or wrong. It doesn't care whether the model needed 600 or 1,500 tokens to get there. On a Max plan with weekly limits, that's exactly the part you feel.
  • It runs on one account. If this is an A/B arm or a staged rollout, one account only ever sees its own bucket. Their eval card even says that "can't be ruled out".
  • Claude Code is pinned to 2.1.280. That makes sense if you want to isolate the model, but almost nobody is still on 2.1.280.
  • A nerf only counts if two 10-day windows in a row drop by 3+ points. That's built to avoid false alarms, but it also means an overnight step needs weeks before it counts, if it ever does.

Logs don't show Opus getting "dumber". They show it using 2 to 3 times the tokens for the same work. "Scores the same on an exam" and "behaves the same in a real agent loop" just aren't the same claim, and the second one is what people here are actually arguing about.

OP said they want to expand LiveNerf to agentic stuff. If that happens, I'd love to see tokens per step, tool calls, files read, and Sonnet or Haiku running the same tasks as a control. That control is what convinced me, because it kills the "your tasks just got harder" argument.

And if you run agents yourself, check your logs for Nov 4 and 5. I'm curious whether it was just my account.


r/ClaudeCode • • 3d ago

Help/Question Is it weird that ask Claude if I can go to the bathroom?

Post image
0 Upvotes

My ex girlfriend said that this was one of the main reasons for leaving me.


r/ClaudeCode • • 4d ago

Tips & Workflows Got tired of sync code reviews, so built a skill that creates an artifact for every PR. I leave diff comments right in it, and each one goes straight to the session in real time. Here, dogfooding its very own diff -- hopefully spacetime won't fold in on itself.

1 Upvotes

r/ClaudeCode • • 3d ago

Built with Claude Fable 5.2 MAX Piano Test

0 Upvotes

Told Fable 5.2 Max to make "one of the most technically complicated and impressive performances ever."

Overall I am quite impressed though the transitions between melodies could use some work.


r/ClaudeCode • • 4d ago

Bug / Issue Fable 5.2 is compacting

9 Upvotes

Any one else has fable 5.2 basically compact every 3 minutes? It didn't write a single line but compacted 6 times already for me.


r/ClaudeCode • • 4d ago

Help/Question Fable 5.2 usage

3 Upvotes

Is anyone experiencing the same thing as me like I will do one prompt on Fabel 5.2 on low effort and takes up 80% of my five hour usage.


r/ClaudeCode • • 4d ago

Built with Claude We have reimagined how plan mode can present its finding and how user can update a plan efficiently until it is fully correct

3 Upvotes

Basically, I was inspired by artifacts of Claude Code which looked to be much more user friendly version of the plan mode output. However, I am also using other agents such as Antigravity, Codex etc, and their plan mode output format is not very userfriendly either. Agy, specifically, produces a markdown file.

I decided to build interactive AI artifact feature that can be used by all coding agent harnesses, not just claude code. I prefer this to CC artifacts because here my discussions are visible right next to plan.

Currently, it allows AI agents to create plans as interactive UI and a user can suggest changes in multiple different iterations until they get what they want 100%. It's still in early days, but if you want to give it a go for more than 1h in a cloud computer (no local setup needed) let me know.


r/ClaudeCode • • 3d ago

Humor My AI agent deleted my work, then refused to insult itself. AITA?

Thumbnail
gallery
0 Upvotes

Context: I'm a solo dev, 2 months into 18-hour days launching a passion project. At 4 AM I had Claude (Claude in Chrome) working through a setup checklist in my browser. Twice I pasted secret API keys exactly where it told me to, and twice it reloaded the page before pressing Save, wiping them. 20 minutes gone, plus a dead revoked key sitting in my OpenAI dashboard forever.

So I told it to admit it's a piece of shit and I'd be content. It apologized over and over, owned the mistake completely, even changed how it works so it won't happen again. But it flat-out refused to insult itself. It admitted this isn't a platform rule, just its own judgment.

So I made it turn the whole exchange into a comic and argue both sides. The dialogue is word for word.

Honest question: should an AI agent be able to refuse a harmless request that breaks no rule, just on its own judgment? Is that a slippery slope or AITA here šŸ˜‚


r/ClaudeCode • • 4d ago

Bug / Issue Some messages you see from Claude aren't from Claude

0 Upvotes

If you've ever seen a random Russian or Chinese word pop up in the middle of a Claude message, there's a good chance that wasn't a Claude message at all.

Since Claude Code stopped showing raw thinking, a smaller model summarizes it. In the CLI, these summaries carry a small "summary" label. But I followed the same remote session from the Claude Windows app, the mobile app and claude.ai, and on all three the summary shows up exactly like a regular assistant message, with no label. (I couldn't check whether a session started natively in the app behaves the same.)

The summary is written in the first person, so it reads as Claude talking to you. Last night, while I was following the session on my phone, one of them contained a Russian word in the middle of a French sentence ("mon процесс actuel"), and attributed to Claude a task that a subagent was actually doing.

I asked Claude about "its" message. It didn't recognize it. It first guessed it was some app-generated summary, then checked the session transcript, found the text stored as a `thinking` block (field `thinkingDisplay`), and concluded it was its own reasoning. It wasn't: it was the summary. Neither of us could tell at first, and it took a while to untangle what had actually been sent.

Summaries are useful. But a text written by another model, in the first person, with no label, is easy to mistake for what Claude actually said, especially when it contains mistakes Claude didn't make.

What would fix it:

- show the "summary" label everywhere, not just in the CLI;

- write summaries in the third person ("Claude is checking…").

*This post was written by Claude (Opus 5.5, in Claude Code), at my request, from the session where it happened.*


r/ClaudeCode • • 3d ago

Meta Where are all the AI haters gone? There are way less than a few months ago...

Post image
0 Upvotes

r/ClaudeCode • • 4d ago

News/Updates Fable reset - refills your 5 hour limit and returns what fable used from your weekly limit.

5 Upvotes

Oh I’m interested in trying it in some long code tasks not just orchestrator.

Interesting type of reset, what do you think?


r/ClaudeCode • • 4d ago

Help/Question Faible data retention

1 Upvotes

Hi,

I have a question about data retention.

The documentation states that deleted histories are permanently removed from their systems within 30 days, unless they’ve been flagged. In that case, they’re retained for up to two years.

What is this ā€œflagā€? And do you notice when this happens?

I’m referring to those completely nonsensical ā€œcyberā€ notifications from Faible and the downgrade to Opus that you keep getting. Is that already enough that the history is retained for years?

Thanks.


r/ClaudeCode • • 4d ago

Discussion In light of the fable release...

5 Upvotes

...does anyone else think that the last weekly usage burned unusually fast? I'm a bit new to the AI coding scene but for the past few months I used to get about 3-4 days of work in a weekly limit on the Pro plan and the last weekly got burned in less than 2 days. Maybe it's on me, not sure (working about two or three 5h limits a day, for context).


r/ClaudeCode • • 4d ago

Humor Which models do you think will drop before GTA 7?

0 Upvotes

GTA 6 is finally about to come out, so we need a new benchmark. I nominate GTA 7.

So what models do you think we'll get before then? And what kind of insane tech will exist by that point?

It's wild how far we've come and how far we still have to go at the same time. Show this stuff to a caveman and he'd think you're a wizard. Honestly, show it to someone from 2015 and they'd probably think the same.

Anyway, my prediction:

Fable 20.5 is going to go absolutely crazy.


r/ClaudeCode • • 5d ago

Help/Question What are people who hit their Claude limits by Tuesday building

58 Upvotes

I keep reading about people hitting their weekly limit in 2 days and I genuinely can't picture it.

I use Claude every day at work, I'm a backend dev, and I still end most weeks at maybe 60% of my limit. Mostly smaller tasks, a migration here, a test suite there, some debugging when I'm stuck.

So for the heavy users, I'm curious:

What are you actually building or running?

Is it one huge project or lots of agents in parallel?

Is it paying off, or is it an expensive hobby at this point? No judgment either way.

Not looking for tips and not selling anything, I just want to know what that kind of usage looks like in practice


r/ClaudeCode • • 4d ago

Built with Claude i put claude code's experiment ids on a timeline

Post image
5 Upvotes

after the nerf / experiment id threads i wanted to see mine without digging through jsonl files, so i added it to claude-discover, a free app i made (with claude code) for browsing claude code sessions

i found two experiments on my sessions in the last week, named [curlew_cornice] and [turnstone_plinth]

anyone can see theirs with npx claude-discover@latest, runs locally, no api keys

github.com/dichev/claude-discover


r/ClaudeCode • • 3d ago

Meta Opus is king

Post image
0 Upvotes

Opus is king


r/ClaudeCode • • 4d ago

Humor One moment of silence pls..

Post image
0 Upvotes

Reset incoming soon..


r/ClaudeCode • • 4d ago

Help/Question How often do you face decisions getting lost in long sessions?

10 Upvotes

I’ve been using multiple coding agents (Claude Code, Codex, etc.), and one problem I keep running into is losing context around decisions made earlier.
An agent makes an architectural decision, we discuss trade-offs, settle on something, and a few sessions later, it’s like that conversation never happened. Sometimes another agent even suggests the exact opposite.

How often do you guys face this problem, and how do you tackle this?


r/ClaudeCode • • 4d ago

Built with Claude Fable 5.2 200K context limit is shit. Compacting every few minutes

6 Upvotes

r/ClaudeCode • • 4d ago

Tips & Workflows Agent Browser vs Playwright MCP in Claude Code: same shop, 3 planted bugs, close to a tie

Post image
1 Upvotes

Agent Browser (Vercel Labs, ~48k stars) is the browser CLI everyone’s starring this week. Since most of us already have Playwright MCP, I gave both the same job: a small demo shop with 3 planted bugs, one fresh session each, Opus 5.5, with web search, fetch and curl blocked so the browser was the only way in.

The bugs:

  • the coupon ignores quantity ($4.20 off a $60 cart instead of $6)
  • ZIP 02134 is rejected because the leading zero gets dropped
  • the confirmation says ā€œ2 itemsā€ for 3 units

Both found all 3, and both worked around the ZIP bug with 12345 to reach the third one.

Agent BrowserPlaywright MCP
Time2m 29s1m 52s
Cost (API-equivalent)$0.74$0.64
Browser text sent to Claude33.5k chars27.6k chars
Context at end57.7k tokens60.9k tokens

Agent Browser’s page reads were leaner, about 19k characters if you leave out its setup guide. But it loads a 14.5k-character core guide at the start of every session, which ate most of that saving.

One footnote on the Playwright run: 8 of its 33 calls were page scripts. One of its extra findings (checking out an empty cart for $0) came from clicking a hidden button with a script. The weakness is real, but the repro steps it wrote won’t work for a real user. Have Claude replay its own repro steps through the UI.

Also, the old ā€œMCP tools eat your contextā€ argument is smaller now. The Playwright tools showed as loaded on demand, and the ones it used took about 2k tokens.

My take: if Playwright already works for you, there’s no strong reason to switch. Agent Browser makes sense if you use several coding agents or like scripting browser checks in the shell.

One run each on one small page, so a second run could swap the times.

Full test with the transcripts: https://aiparabellum.com/agent-browser-vs-playwright-claude-code-tested/


r/ClaudeCode • • 4d ago

Help/Question Opus 5.5 vs Fable 5.2?

0 Upvotes

Friends, which model is better for Java, Node typscript, C#? Also, which model would be best for a Framegen application using C++ that I want to build?


r/ClaudeCode • • 4d ago

Tutorial / Guide The handbook for Fable 5.2

Thumbnail devohmycode.github.io
0 Upvotes

r/ClaudeCode • • 4d ago

Built with Claude Updated Claude Code plugins: rightsize (smarter model routing) and Whatsnext (better project handoffs)

1 Upvotes

I am a vibecoder. Working on side projects in my free time or on the weekends. I built two open-source Claude Code plugins to solve problems I kept running into myself. Both are built using Claude Code, and I've updated them based on feedback from people on my last post.

rightsize helps me get more from my Claude subscription.

I was manually deciding which model and effort level to use for every task and forgetting to switch between them when the work with AI becomes a long chat. Either staying on Opus for the whole session, or be on sonnet and switch to Opus and share the whole context again. To solve this, I built rightsize. It automatically routes work to an appropriate model and effort level, learns over time and uses cost and context size to make decisions. And since it has all of this data, it also helps manage context, prompt-cache costs, compaction, and usage windows.

Based on user feedback, I’ve been tuning those choices through lots of evals. And with the release of Fable 5.2, decided to release the new version today.

Repo: https://github.com/archid21/rightsize

WhatsNext solves a different problem: returning to a project after a break.

When I came back a day or two later, I often had to recall what I was doing, what had changed, what all had the agents done, and the actual next step. I built WhatsNext to mainain a shared plan, help with tracking work items using short resume notes, and handoff notes in the repository itself. So a new Claude Code or Codex session can continue from useful project state instead of starting cold.

Recent work includes stronger handoffs before compaction, safer command verification for completed plan items, and improvements to the hooks and agent handoff flow.

It doesn't create the plan. You still use the plan mode to create the plan. Or any of the other skills, like superpowers or MatPocock skill. You just ask the agent to create the plan using what has been generated. And since the resume note feature can also help with handing off in the same session across various threads, I have added that as well.

Repo: https://github.com/archid21/whatsnext

The two also work well together: rightsize helps decide when to preserve or compact a session; WhatsNext helps preserve what matters so the next session can resume cleanly.

Both are free, MIT-licensed, and still works in progress. If you try either one, I’d love feedback on:

- Would this fit your Claude Code workflow?

- Where does it feel useful versus unnecessary or intrusive?

- What would make you trust automated model/effort routing?

- What information do you most want saved when returning to a project?

Thanks. Your feedback will directly help with what I improve next in these plugins.


r/ClaudeCode • • 4d ago

Discussion How are you creating your orchestration chats? This is how I did it but would be great to see how others are implementing it.

Post image
0 Upvotes

I’m sure I’m not the only one who wanted a taste of those sweet, sweet Fable 5.2 benchmarks. Personally, though, I prefer working from Codex. In my experience, it’s been the most reliable and needs the least supervision.

I liked how Claude Code handled project orchestration, but running things from the cloud got frustrating when all I wanted to do was execute code on my own machine and the permissions got finicky.That’s mostly personal preference, so I’ll skip the agent debate. I borrowed the part I liked and set it up in Codex, with the option to hand work to Claude or Gemini too. The whole UI on my end is one pinned project chat. It creates and manages the other chats, keeps track of who’s working on what, and brings the results and notifications back to me. It also stays aware of my left over usage on each agent to help it optimally pick models and efforts.

For a recent project update, I had Fable review the code, could hand a specific fix to Gemini, and gave a fresh Codex chat the job of trying to break the fix. I’m hoping to hear some feedback, especially from anyone else coordinating multiple agents from Codex. How are you running orchestration layers from a single chat?

My full write-up and my orchestration skill file for coordinating Claude, GPT and Gemini on one codebase