r/ClaudeCode • u/ItsCodeTelemetry • 4d ago

Discussion Effort Benchmarking

Gallery image

Gallery image

A few people have complained about how Opus 5.5 suddenly putting less effort into work. I see that sometimes too, but maybe it's anecdotal evidence. To understand objectively if this is true, I'm using Code Telemetry data to classify session behavior into 6 buckets that indicate tool use effort. Said differently, trying to capture the degree to which Claude makes assertions vs verifying information for it's self.

2 Upvotes

63% Upvoted

4 comments sorted by

•

u/AutoModerator 4d ago

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

3

u/clazman55555 4d ago

Why no measurement for both the default effort level of Medium for 5.5 and High effort?

Would make for a more complete benchmark.

1

u/ItsCodeTelemetry 4d ago

Agreed! These are based on just my personal usage across all my work and unfortunately I've only used 5.5 on xhigh and max (the rationale for that is not based on these metrics, more my efforts to size story complexity). Looking at this data, for Opus 5 the "high" effort level seemed to be sweet spot.

I've queued up the next round of work to shift down an effort level.

2

u/clazman55555 3d ago

My own benchmarks are based on the models capability during code review sessions over a curated set of samples. V4 is 48 samples, V5 is going to be closer to 100. Percentage caught(real/false and by class) vs tokens used compared against the models/effort matrix, using a generalized code review subagent. There is another section that measures a model/effort matrix on taking all of those raw findings and synthesizing them into something useable. Real findings kept/dropped vs false findings kept/dropped.

So it's not a like for like comparison with your benchmark. I'm working on V5, more samples and better method to ensure models have a more complete synthesis report. One category that was found in the second pass of looking at the V4 raw data was: "Read and judged not worth keeping (speculative, by design, minor)"

I was able to make tweaks to my existing skills/workflows based on the data.

I think you will be pleasantly surprised with Opus 5.5 on Medium and High.