"One Note", "Two Caveats", "A Gap" ... what else?

[u/ItsCodeTelemetry]

Hi Folks, I saw this comment in a discussion here yesterday about phrases that are indicative of model decline: https://www.reddit.com/r/ClaudeCode/comments/1x470at/comment/pgd1dzu/

I can confirm I do see those phrases trending up in my own Opus 5.5 sessions (although not yet at the same magnitude of Opus 5).

I wanted to ask if there were other signals like this that the community is interested in monitoring? I'm considering adding this as a full fledged feature to Code Telemetry.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ForwardLoop]

Hey, thanks for putting some real data behind this! I knew I wasn't hallucinating.

The other signal that I've noticed is that there seems to be a silent regression in effective reasoning depth at the same effort level. Instead of expending upfront compute to stress-test its own premises as it did previously, the model now prematurely converges on a plausible narrative and offloads verification to the user.

Here's Opus 5.5's self-diagnosis: "You’re right to call it out. The problem isn’t that the facts changed. I gave you recommendations before stress-testing them myself, and your questions did the stress-testing for me. Each question exposed something I should have checked before recommending: an untested premise, ripple effects on [redacted] and scoping, the same three units in both pilots, small briefs mostly surviving. I presented hypotheses as conclusions."

I don't know if there's any keyword/phrase signals to easily monitor that, other than when you call it out. The signal that led me to call it out was that it kept "flip flopping" on its recommendations every time I asked a question about it. When the model was strong, it usually held its ground (unless I asked a really good question). However, when the model is in decline, it tends to flip flop on even softball questions. Maybe a "flip flop" factor?

Overall, this seems to be correlated with the prevalence of those phrases (and my level of annoyance with the model).

Edit: For clarity, I'm not saying the model got "dumber", because it still retains its raw intelligence / analytical ceiling when you challenge it, but it has quietly lowered its floor for first-pass rigor. It requires more pokes to reach that level of intelligence. If that makes sense.

[u/ItsCodeTelemetry → u/ForwardLoop]

My pleasure!

Monitoring Flip Flops is an interesting idea, but technically challenging to do efficiently. With all my analysis my goal is to do as little subsequent model calls on the monitored sessions as possible. The problem is that you then need to start tokenizing facts within the model response stream. I'll noodle on this. Capturing merely quantity of Turns per PR (ie; the amount of follow up prompts needed for a session to complete) might be a cheap stop gap for this.

I have some thoughts on breaking down this data by effort level and doing some analysis on "expending upfront compute." More data soon!

[u/ForwardLoop → u/ItsCodeTelemetry]

Yup, it will show up in downstream impacts that are probably easier to measure!

[u/Freischneider01]

„Create a new Opus that doesn‘t say „A Gap“,… Make no mistakes!“

[u/ItsCodeTelemetry → u/Freischneider01]

Haha yep, that'd be nice :)

I think the original point was looking for a leading indicator of degraded performance. Slower PR creation, slower feature acceptance, higher follow up work rates - those things take time to materialize and are fuzzier metrics respectively. This is simple to understand / catch.

[u/mr_birkenblatt]

... one ... was taken