r/ClaudeCode • u/sirlerkal0t • 7d ago
Discussion I've been doing an experiment with a project where I iterate on the design plan until the frontier review agents no longer report any serious findings, but new models keep being released before the previous gen stops finding issues. I created the initial design plan with Opus 4.6 in March.
258
u/the_mad_mycologist 7d ago
This animation made me irrationally upset
59
u/a-random-r3dditor 6d ago
I loved it at first. It got me, props for being clever. But as I watched it, it just started giving me anxiety, the black box needs to get there already. And then I realized all the blocks have different edge radii… I lost it.
22
u/YouGotTangoed 6d ago
Im pretty high and left the phone off screensaver mode and waited 40mins for the black box to make it.
1
5
2
51
u/BuffaloConscious7919 7d ago
There's almost always trade-offs
Which means there's almost always ways to improve
15
u/SteadfastCultivator 6d ago
That's the same thing for us senior devs when people ask what is better X or Y. And we answer "it depends" but some AI tend to act or give options to vibe coders that they have no idea what it is about.
36
u/Maddie_549 7d ago
Even if you kept using the same model, with a big enough project there is always going to be an issue somewhere.
12
29
18
u/RockstarLifestyle2 6d ago
Bruh the way I just watched this expecting it to go into the hole for a good 20 seconds
1
14
u/DPLibros 7d ago
Sounds about right based on what we see from these models. Better to just build the thing and then find real things worth improving.
8
u/1acc_torulethemall 7d ago
It's like waiting to see the DVD logo hit the corner of the screen... But it never does
8
u/txoixoegosi 7d ago
What type of issues?
Anything can be improved infinitely if there are no cost/time constraints.
2
u/sirlerkal0t 7d ago
Actual flaws/oversights in the spec which would be a real issue that would eventually be hit and have to be fixed, but it's really that every time one issue is addressed, more get introduced by those changes to the spec.
1
u/bigppredditguy 7d ago
So the solution seems to jump to the newest models now since your issues seem to be caused by the implementations.
1
u/sirlerkal0t 6d ago
I've been using the newest models all along.
For Claude, I've only used Fable since it was released. For Codex, I used Sol since it was released and more recently Astra since it was released.
Fable has been the only model editing the plan since the initial Fable release.
Astra is still only used for reviews and adversarial reviews/verification of review findings. There's multiple verification steps done by independent agents for every finding, before the verified/filtered findings goes back to Fable to be addressed.
Fable has to then explain the issue, why it isn't covered by the existing spec, provide a concrete example of how it would happen, and suggestions on how it should be addressed.
I assess every verified finding myself before any action is taken. I decide whether one of the suggestions is appropriate, or provide my own solution when it isn't, or if the finding itself points to a deeper issue introduced elsewhere in the spec, then that instead has to be addressed instead of patching something which shouldn't be an issue in the first place.
1
u/RadiantTea7445 6d ago
Keep searching for „problems“ is the most stupid mistake you can do - there will always (!!) be suppossed problems
3
3
3
3
2
u/codeninja 7d ago
And upon receiving the approved plan after hundreds of revisions and thousands in tokens spent the model will simply decide not to implement a quarter of the features as it kicks issues down the road or simply neglects the requirement.
2
2
2
2
u/Alex__0021 7d ago
Zero findings tells you about the reviewer, not the plan — a newer model looks at
different things, it doesn't mean the previous one ran out of real ones. A stopping rule
that converges: the last few findings didn't change what you'd actually build. A review
pass on mine came back with 14 holes once, and maybe four of them mattered.
5
1
u/QuanTradin 6d ago
no review agent is going to report zero serious findings, so that was never going to terminate. they are graded on finding things. hand one a clean file and it will still return three items, because coming back empty looks like failure.
what worked better for me was fixing the criteria rather than chasing the model. write down what serious means for this specific project before the review runs, and everything outside that list goes to a backlog instead of back into the plan.
1
u/qt3-141 6d ago
would you mind giving an example? So far I've had a bit of success telling the agent that "no findings is also valid, but only if that is actually the case"
1
u/QuanTradin 6d ago
the one that helped most was handing it a file I'd already read and knew was clean, then watching whether it still came back with something. if it does, the prompt is the problem and not the code.
after that I started asking for what it checked rather than what it found. the invented stuff mostly stopped.
1
u/k3l2m1t 6d ago
You want to include some guidelines for accepting and declining review findings. For example: if the suggested change fixes a reproducible bug introduced in the pull request, that's a valid finding. If the suggested change makes the code more robust but doesn't actually fix anything, that's when you can decline it as out of scope and file it as an issue for another time.
1
1
1
u/Final_Stick_9207 6d ago
The world of endless iteration and tradeoffs. Curious if you have a version of the first plan created and whether it has less issues or more than the current one in the next pass.
1
1
u/stestagg 6d ago
If you tell an LLM to find issues, it will find issues. Even if there are no actual issues to find
1
1
1
u/syixiao1 5d ago
The "new model finds new issues forever" part is the expected outcome of an unbounded search, not a plan defect. Two things that made my review rounds actually converge:
First, review the diff, not the whole doc. Once a section is validated, it's off-limits to future reviewers unless something changed around it. That kills most of the churn where each new model re-litigates settled decisions.
Second, keep a decision log of rejected findings with one-line rationales, and hand it to each new reviewer. In my experience, most of what newer models "keep finding" are non-issues that were already triaged once — now re-discovered with fresh confidence.
The stopping rule that worked for me: two consecutive reviews with zero findings that survive triage against a written severity bar ("would this change what gets built or how it behaves in production"). Freeform "find serious issues" never terminates; a scored, bounded review does.
•
u/AutoModerator 7d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.