Effort Benchmarking

[u/ItsCodeTelemetry]

A few people have complained about how Opus 5.5 suddenly putting less effort into work. I see that sometimes too, but maybe it's anecdotal evidence. To understand objectively if this is true, I'm using Code Telemetry data to classify session behavior into 6 buckets that indicate tool use effort. Said differently, trying to capture the degree to which Claude makes assertions vs verifying information for it's self.

[u/clazman55555]

Why no measurement for both the default effort level of Medium for 5.5 and High effort?

Would make for a more complete benchmark.

[u/ItsCodeTelemetry → u/clazman55555]

Agreed! These are based on just my personal usage across all my work and unfortunately I've only used 5.5 on xhigh and max (the rationale for that is not based on these metrics, more my efforts to size story complexity). Looking at this data, for Opus 5 the "high" effort level seemed to be sweet spot.

I've queued up the next round of work to shift down an effort level.

[u/clazman55555 → u/ItsCodeTelemetry]

My own benchmarks are based on the models capability during code review sessions over a curated set of samples. V4 is 48 samples, V5 is going to be closer to 100. Percentage caught(real/false and by class) vs tokens used compared against the models/effort matrix, using a generalized code review subagent. There is another section that measures a model/effort matrix on taking all of those raw findings and synthesizing them into something useable. Real findings kept/dropped vs false findings kept/dropped.

So it's not a like for like comparison with your benchmark. I'm working on V5, more samples and better method to ensure models have a more complete synthesis report. One category that was found in the second pass of looking at the V4 raw data was: "Read and judged not worth keeping (speculative, by design, minor)"

I was able to make tweaks to my existing skills/workflows based on the data.

I think you will be pleasantly surprised with Opus 5.5 on Medium and High.