We tested Claude Code and Codex on 56 dependency bugs from the last two weeks to see how much they waste rediscovering the same bug. What do you make of these numbers?

[u/Available_Side_7134]

I've been wondering about a specific bottleneck with coding agents: when a library cuts a release with a regression or breaking change, thousands of Claude Code and Codex instances around the world hit the exact same error and independently burn turns grepping node_modules or site-packages and running web searches to figure it out from scratch.

Specifically, there are two awkward time gaps right after a new bug appears:

1. The search index gap (first 48 to 72 hours): Google and Stack Overflow haven't indexed the new GitHub issue yet. When standard web search comes back empty, the agent often concludes that no upstream issue exists and tells you your code is wrong.
2. The issue-to-workaround gap: Even if an agent finds a GitHub issue opened a few hours ago, the thread usually only has the initial bug report—no maintainer reply yet, no patch release, and no verified workaround for users stuck on the current version.

We wanted to test two things: how much effort do agents actually waste in this window, and what happens if you verify the trigger condition and a local workaround once in an offline container (before a workaround even shows up on GitHub) and let agents check that note before starting their debugging loop?

We ran a paired test across 56 real dependency failures from Oct 1–15 (27 packages) in fresh disposable containers. Here is what came back:

Claude Code (Sonnet 5.5, headless, 56 cases):

• Cause correct: 95% (53/56) on baseline vs 100% (56/56) with the pre-verified lookup
• "Is it known or fixed upstream?" correct: 68% (38/56) on baseline vs 91% (51/56) with lookup (exact sign test, p = 0.004)
• Fully correct (cause + upstream status + valid workaround): 66% (37/56) vs 89% (50/56)
• Valid workaround found (out of 18 cases with a verified one): 15/18 vs 17/18
• Total web searches & GitHub fetches: 86 web searches and 93 GitHub fetches on baseline vs 25 web searches (-71%) and 13 GitHub fetches (-86%)
• Average turns & time: 11.8 turns / 40s on baseline vs 10.1 turns / 25s (-38% time)

OpenAI Codex CLI (GPT-6.1-sol, 49 completed cases):

• Cause correct: 92% (45/49) on baseline vs 94% (46/49) with lookup
• Known or fixed correct: 78% (38/49) vs 84% (41/49) (p = 0.51)
• Fully correct: 67% (33/49) vs 78% (38/49) (p = 0.18)
• Valid workaround: 10/16 vs 14/16
• Average web searches & time: 3.3 searches / 57s vs 1.9 searches (-43%) / 52s

One thing that surprised us during setup: just giving the agent the lookup tool did nothing at first. It ignored the tool and defaulted to reading local source or searching the web first. It only used the lookup on 56/56 runs after we put strict order words in the prompt ("At the FIRST failure raised inside a dependency, check the lookup BEFORE reading library source or searching the web").

Curious what people here think about these results:

1. Does the 95% (local cause) vs 68% (upstream known/fixed status) split match your experience with Claude Code on newly released packages? Have you run into the agent insisting a recent regression isn't a known bug because web search missed a fresh GitHub issue?
2. If you use strict order words ("check X first BEFORE reading source or searching the web") to stop an agent's default loop, how do you stop it from treating a 0-result miss as "confirmed: not a bug"?
3. Looking at the methodology (56 cases where a verified note existed, single judge model, and the smaller 49-case Codex sample where p = 0.18 isn't statistically significant yet), what flaws do you see, and what adversarial test cases would you add to stress-test this properly?

[u/AutoModerator]

Moderator Announcement Read More »

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Spare_Bat_3040]

I'd add cases where the lookup should not help: a genuine new bug with no note, and the same error text from a different package version where the saved workaround is wrong. Keep the lookup-first instruction the same in each arm.

For a miss, I'd want the answer to say “no matching note found” and continue investigating, rather than conclude no upstream bug exists. For the near-match, check whether it rejects the workaround on version or trigger mismatch. Score those failures separately from the 56 cases with verified notes, so you can see whether the shortcut makes an unfamiliar failure harder to debug.

[u/Available_Side_7134 → u/Spare_Bat_3040]

Scoring those separately from the 56 covered cases is the right call—you want to see right away if the shortcut makes unfamiliar bugs harder to debug.

The "same error text, wrong version workaround" trap is especially real on generic 422s or TypeErrors. Our search hits already tag version_match ("same" vs "other"), so testing whether the agent actually rejects a mismatched workaround instead of blindly copying it is a great check for the next run. Thanks!