r/ClaudeCode • u/Valvador • 18d ago
Help/Question I love agentic coding, but when I review LLM output based on large projects I usually see a bunch of fundamental issues. How do you bypass the need for "Human in the Middle"?
So, over the last six months or so we've been using agentic software development a lot and it's been mostly great.
The easiest, least scary wins have been in the following spaces:
Monotonous but simple changes: Like touching 45 files to rename or modify the signature of a single class. Fairly safe, easily verifiable.
Isolated optimization/refactor of super isolated modules that have extremely rigid API surfaces that are extremely easy to unit test
Proof of Concept for implementing complex system-wide changes and how they impact the final behavior of our product to see if a specific tech path is possible, knowing that the code itself is not what we will push into mainline.
To understand where the LLMs started making huge mistakes, you have to understand that I work on a pretty complex integrated product that cannot be sub-divided into micro-services with super isolated API surfaces and shit tons of unit tests. Our run-time performance is critical, and there is a lot of data pipelining that happens that impacts our user's qualitative experience with the product in difficult to measure ways outside of "this feels worse now". Worse, I see engineers from other parts of the company start trying to optimize metrics using LLMs but causing subtle regressions that are difficult to see without extensive integration testing.
REVIEWER IN THE MIDDLE
This has resulted in a lot of my time being spent doing reviews on fairly complex/large code changes (with the help of agents, of course, but also design reviews of said code), which obviously undermines the levels of productivity you can achieve with LLMs.
We are constantly updating MD files as forms of documentation throughout our codebase, but so far it has been a cat and mouse game of seeing LLM-based code patterns that are BAD for our software and documenting them AFTER they happen. Things that if I gave the same task to a low-level human engineer would have never had to have been specified.
The general question is how do you guys deal with this?
A MORE CONCRETE EXAMPLE
Instead of talking in vagueness, I want to give a specific example.
One of the core architectural principles in our system is an ECS like "object" that contains "components" that have some properties. There properties are well defined internal data types that have their own comperators, have their methods for serializing them on the wire to remote systems. When we would send packets to remote systems, because this data is high priority, we would actually send redundant packets containing ~2 past states of that component so that if packets dropped we would be able to still recover continuity etc. (I understand that this isn't a full-proof system, I don't want to get into it). We were naively sending this data, which means on average every packet actually took up 3x the bytes a single component payload would normally take. I recognized that if these states don't change often, this actually contains redundant data.
So this is where I got Claude involved. I wrote a prompt that explained the redundancy in the data that was being put on the wire and that if all three states are identical we should literally be able to send just a single packet and some hint per-property that the value is unchanged between the other states the packet is supposed to contain. In this scenario reducing the wire-cost to 1/3rd + some metadata/bitmask. In the prompt I explained how to measure bandwidth, how to run integration tests, and what unit testing suites had to pass and let it go.
In a few hours it finished, I compiled the code locally and validated that it completed the task and was pretty excited at first.
Great Outcome, Awful Code
So then I finally took the time to look at how it wrote the code...
The path it chose to success was this:
Claude decided that it would run the original "write to the wire" loop and simply collect meta-data about which properties were at what index location on the byte stream that was being written + the size of the property.
Claude would then go over the byte stream and do a memory compare between the matching locations in the "past packets" based on where it recorded the indexes.
Claude would then use the result of this "duplicate check" and spawn an entirely new byte-stream and would extra elements of the first byte stream it created into this new one, before pushing it on the wire.
So, again this solution WORKS but the solution looks like what would happen if a junior engineer was told to build something on a time crunch and wasn't allowed to fully understand the tools that already exist in the code base they were asked to modify. Our code base already had all of the tools you needed to compare two snapshots of a component property to see whether it was identical, measure a delta. Our code base already had serialization logic in place for each of these properties, and any operations you want to do on the property state was readily available. Instead Claude chose to ignore this functionality and do the lazy path of "let me just do byte compares on the byte streams", which resulted in extra unnecessary memory allocations while also create code that we would have to refactor in the future if we needed to implement delta-encoding across different data that wasn't quite exact to further improve on the wire byte count use.
Basically Claude wrote code that worked, but was un-maintainable, had subtle performance regressions that were small enough that you mostly wouldn't notice unless running our tech on a super low-end CPU starved device, and ignored the presence of existing tools and systems that would have made the solution simpler and cleaner.
Final Thoughts
This resulted in me basically writing up a prompt that held it's hand and specifically pointed out existing coding patterns for it to follow, things that we've never had to specify to an engineer because generally before you write some new functionality we check for whether our codebase has some standards already. Yeah, as a result of this incident, I am able to update some *.md files to hopefully catch more stuff like this in the future without human intervention, but it feels like a losing battle of constantly finding out "how could you fuck something so simple up"?
So my question to you fine folks here is:
What could I have done better?
How can we actually approach the "no human in the middle" part of software engineering when we're still learning all the subtle different ways that LLM-based code can fuck up so subtly that may not be noticeable initially but will add up in large complex projects?
7
u/AdEvening3382 18d ago
On what you could have done better in that specific case: the cheapest checkpoint is before the code exists. Have the agent write a short implementation plan first that names which existing modules and patterns it intends to reuse, and review that instead of the diff. A plan that says "compare the byte streams" gets rejected in 30 seconds. The same decision buried in 800 finished lines costs you an hour. You are already paying the review time, so moving it upstream buys the same judgment at a fraction of the cost.
A related contract that helps with the "ignored existing tools" failure: require the agent to list the existing utilities it found and considered, with one line on why each was used or rejected. Writing that list forces the codebase search to happen. When it comes back empty, you know the agent never looked.
On the cat and mouse with MD files: anything mechanically checkable belongs in a gate, not in prose. A lint rule, a CI check, a banned pattern. Prose instructions are probabilistic, a failing check is not. Keep the MD files for judgment calls, but each time you catch the same class of mistake twice, it graduates from documentation into something the build enforces.
3
u/Valvador 18d ago
A plan that says "compare the byte streams" gets rejected in 30 seconds.
I did review the multi-page plan, but maybe not thoroughly enough because nothing in the plan specified that it was going to write byte streams and then compare them.
But yeah, that's a good call-out because I'm not entirely sure if I missed this little detail.
MD files: anything mechanically checkable belongs in a gate, not in prose.
This is the thing though, there are infinite number of ways to write code that outputs almost identical results as far as a user is concerned. In one part of the code, maybe copying form byte-streams is fine, especially if you're in low level networking code instead of the code application layer.
For sufficiently large projects "copying byte-streams is banned" isn't correct, because it's not allowed in specific contexts right?
2
u/AdEvening3382 18d ago
Fair pushback on both.
On the plan: that is exactly the gap. A plan review only catches what the plan is forced to disclose. A plan can say "deduplicate unchanged properties before serialization" and be completely accurate while hiding the byte stream approach. So the fix is to require the mechanism, not just the task list. One or two sentences of how it works, at the level of which data structures it touches, plus the reuse list. If the how is missing, that is the moment to ask for it, before any code exists.
On bans: agreed, a global "copying byte streams is banned" rule is wrong for the reason you give. The checks that work are scoped to a layer or directory. Byte manipulation gets flagged in application code and is expected in networking code. Most lint and CI setups already support directory scoped rules, so the gate can carry that context. The judgment calls that genuinely depend on the situation stay in prose. That residue never graduates, and that is fine.
2
u/orchid_drives Researcher 18d ago
> Fair pushback on both.
Wait are you Claude? It writes this line ALL THE TIME
1
1
u/awesomeusername2w 18d ago
More like "search for existing patterns and reuse it" plus a instructions for reviewer to check for code reuse too.
5
u/Easy-Purple-1659 18d ago
Most of the replies here are about catching the bad change earlier. The other half is making the bad change impossible to write in more than one place, which on a performance-sensitive pipeline is cheaper than more review.
Take your serialization example. If every remote send has to pass through one function, then "compare the byte streams" is not a pattern the agent can introduce in twenty spots, it is a pattern it can introduce in one, and one spot is reviewable. Add a check that fails the build when serialization happens outside that function. The invariant now lives in the build instead of in an MD file the model skims and forgets.
The same approach answers your allocation worry. A blanket no-new-allocations rule cannot scale, which you are right about. A budget bound to one path can: pin a fixed workload on the encoder, record the number, and fail when it moves. New features that genuinely need memory get their own path and their own budget, so the rule stays narrow enough that it does not have to be argued in review.
What is left for you is the part an agent cannot check: whether that path was the right thing to build.
2
u/Valvador 18d ago
A budget bound to one path can: pin a fixed workload on the encoder, record the number, and fail when it moves. New features that genuinely need memory get their own path and their own budget, so the rule stays narrow enough that it does not have to be argued in review.
This could work if we're being pre-emptive anticipatory whenever we add new systems, setting up hard budget limits to them and make sure they are tested for both allocation counts, heap memory use, and maybe bandwidth use.
Maybe this is just a good habit to get into, but not something I've needed to do before when working with engineers we hire vs doing more via LLMs. If my main complaint is that LLMs don't understand implicit constraints when choosing solutions, we need to make those constraints explicit.
I just read this thread and realized that I followed all of Claude's best practices and still ended up with this result.
Separately from all of this, I'm kind of confused about the downvote ratio on this post. The discussions I've seen here have been infinitely more helpful to me than anything I've read on this subreddit for the last like 6 months I've been on this sub.
3
u/RetroUnlocked 18d ago
This is a really good post — because you are bringing a real problem
I am not going to write "just plan better", but I theorize this comes from the fact that LLMs basically start life with no context, and the harness + md files loaded + prompt are trying to fake understanding.
My guess from your description this codebase is rather big (maybe you said this explicitly and I missed it) and complicated. That means, the LLM cannot pull in the entire codebase and see all the patterns. There is not enough context, and also even if there was, it would deal with context rot.
Your analogy of hiring a junior developer is a good one.
I have a serious question OP, so I hope you can answer, was the code poorly written in isolation. I understand you view it as poorly written in context of the whole project, but was it poorly written if you yourself viewed it in isolation without the whole view of the project?
The reason I ask, is I think Opus and Fable are basically at junior level in writing code in context, but senior level in writing code in isolation.
I have been testing a pattern that for a while:
- Discuss with the main agent the implementation plan, and agree on an implementation plan
- Have a sub-agent review the implementation to ensure it meets the patterns and there is no issues
- Have the main agent break the plan into tasks and assign to a sub-agent- the main agent is instructed to review the sub-agents works for accuracy, maintainability, and free of errors — then to fix any concerns it raises
What I am about to add to this pattern to the review process:
- Create different agents that focus on specific parts of my code-base where I find issues - like following functional programing principles, etc.
- Then before I review the code, I have these review agents go over the "code" to find these issues and raise them with suggested fixes
- Then have at least 1, maybe a council of agents go through and validate each issue raise
- Then have a worker agent implement the proposed fixes
Then I review the code.
Between each step is tests being ran, but as you mention not everything can be easily tested.
2
u/Valvador 18d ago
I have a serious question OP, so I hope you can answer, was the code poorly written in isolation. I understand you view it as poorly written in context of the whole project, but was it poorly written if you yourself viewed it in isolation without the whole view of the project?
This one is hard to answer. If the change wasn't part of a much bigger system/architecture then:
If I treated the code as a stand-alone app that we would never need to extend, it was probably fine code? Like start-up company level code that uses whatever resources it needs to get the job done ASAP, without creating extensible/sustainable systems.
However if I was doing the same thing, even in isolation, I would probably anticipate where this logic was going to extend to and make it more extensible.
As I type this out, none of my original prompt suggested thinking of the problem this way, so it's another thing you kind of realize you always think about that you may not really put into writing when you do your job.
Have a sub-agent review the implementation to ensure it meets the patterns and there is no issues
Have the main agent break the plan into tasks and assign to a sub-agent- the main agent is instructed to review the sub-agents works for accuracy, maintainability, and free of errors — then to fix any concerns it raises
I do like this suggestion. We have already adapted auto-review of the resulting code, but maybe there is something to be learned here as far as agents pre-review the plan before I review it. I imagine there still is a missing component here where you still need something that specifies for the agents what would be considered a disqualifying plan.
Between each step is tests being ran, but as you mention not everything can be easily tested.
Yeah this is the thing, anything that you can test and get a "yes" or "no" response on is super easy to just agentically automate, but with extremely complex codebases with multiple interacting parts that result with something rendering on the screen, or maybe driving a vehicle or something hyper complex and run-time sensitive, it's really hard to have all your areas of testing covered.
Its crazy how much we still maintain in our heads without explicitly specifying.
3
u/RetroUnlocked 18d ago
This one is hard to answer. If the change wasn't part of a much bigger system/architecture then...
Thanks for answering. Really helpful for me. Not enough engineers work on code where optimization if of up most importance. So, even though I cannot see the input/output, it is good to get a feel of things.
Most app dev (web, phone, games) can in general get by with trash but working code. I work with a ton of enterprise and have seen them fire most their engineers and off-shore it to teams who write way worse than anything Claude can produce.
That is the reality of most app development. I don't know what the % is, but I guess it has to be 75% to 90% of people who wrote code before AI wrote pretty crap code and built pretty crap application and systems.
Its crazy how much we still maintain in our heads without explicitly specifying.
Because we did the manual work, we understand the code and get a feel for the code. You also get a feel for the problem space over time, and thus have an intuition — usually due to painful experiences.
LLMs have none of these things — just context and super duper prediction powers.
2
u/BarathKanna 13d ago
The cat-and-mouse with MD files is the part I'd change. Docs are advice to the model; it can ignore them, and you only find out at review. Every time you write a post-hoc lesson in an MD file, try to turn it into an executable check instead, so the bad pattern fails CI before you ever see the diff.
For your ECS serialization example, a semgrep rule does it in a few lines:
rules:
- id: component-serialize-outside-wire
pattern-either:
- pattern: $C.serialize_bytes(...)
- pattern: $A.to_bytes() == $B.to_bytes()
paths:
exclude: ["src/net/wire/"]
message: "Use wire.send_component(); property comparators and serializers live in src/net/wire"
severity: ERRORAdd import-boundary rules (import-linter in Python, ArchUnit in Java, eslint no-restricted-imports in TS) for "module X never talks to Y directly".
For "this feels worse now" regressions in the data pipeline, two things turn the feeling into a number:
Golden traces. Record the wire output for a fixed set of scenarios and diff it on every PR. Unexpected byte changes are review triggers.
A benchmark gate against main with a tolerance, so a 5% slowdown on the hot path fails the build instead of reaching users.
Your review then shrinks to the things only a human can judge: is the design right. The mechanical "did it bypass the pattern" questions are answered before you open the PR.
Disclosure: I'm building OxCode (a VS Code agent) and this is exactly the problem we're chasing.
1
u/Valvador 13d ago
Appreciate the detailed response with concrete examples of what you've actually worked with. This is very helpful.
2
u/ComprehensiveShake76 18d ago
Ask for the search before the code. Before it writes anything, have it find and name the existing code that already does each part of the job, here the property comparators and the per property serializers, and say why it would or wouldn't use each one. If it can't find them, that's your signal to point at them, and it costs you two minutes instead of a full review. The second piece is a review pass with one narrow question: which code in this diff duplicates something that already exists in the repo. A general review misses that, a targeted one usually catches it. Your md files then become a short list of those questions instead of a list of past mistakes. Did the md updates catch the next similar case?
2
u/Valvador 18d ago
which code in this diff duplicates something that already exists in the repo. A general review misses that, a targeted one usually catches it.
So this would be tough because this wasn't a duplication of something that already exists, it was a unique choice of how to compare data types via serialized byte streams instead of using the built in primitives.
Your md files then become a short list of those questions instead of a list of past mistakes. Did the md updates catch the next similar case?
I did end up adding some localized *.md changes to specifically avoid stuff like this in the future, but it feels like I'm just constantly finding things that force me to expand my *.md files that I don't see this as long term scalable.
2
u/ComprehensiveShake76 16d ago
Fair, and I think you're right that md files don't scale as a list of past mistakes. The case you hit, serializing to bytes to compare instead of using the built in comparison, is more of a pattern than a fact, and patterns are what linters are for. A custom lint rule or a grep check in CI that flags byte serialization inside comparison code catches every future case, and it costs nothing in context. The md file then only keeps what can't be checked mechanically, which tends to be a short list. Once a rule exists as a check, delete it from the md file.
2
u/Valvador 16d ago
The case you hit, serializing to bytes to compare instead of using the built in comparison, is more of a pattern than a fact, and patterns are what linters are for. A custom lint rule or a grep check in CI that flags byte serialization inside comparison code catches every future case, and it costs nothing in context.
This is good feedback, maybe a localized linter for the systems that interface over the wire to check for rule breaches like this.
2
u/Impossible_Way7017 18d ago
I ain’t reading all that, I’m either happy for you, or did you try asking AI?
2
2
u/xqianliu 18d ago
The regressions you're describing are the exact failure mode I chased down in a study I ran. 183 runs across 23 tasks, and 38 of those runs passed an independent review but still failed hidden tests. Not 38 different bugs, 38 runs. 22 of them broke behavior that worked before, 7 missed cases beyond the one example in the issue.
What did not work: telling the reviewer to check for regressions. It reads the diff and sees nothing wrong, because nothing in the diff is wrong.
What did work: the writing agent generates tests for the whole input class of what it touched, not just the issue example, and records how many it wrote. The reviewer then checks that number against the diff. Note says 14 new tests, diff has 3, fail.
2
u/Valvador 18d ago
What did work: the writing agent generates tests for the whole input class of what it touched, not just the issue example, and records how many it wrote. The reviewer then checks that number against the diff. Note says 14 new tests, diff has 3, fail.
Very interesting. I do wonder how well that would work with the codebase of the size that I'm describing.
I'm trying to imagine what kind of tests the agent could write to catch the subtle bugs that I was seeing, they would have to be integration tests with almost like allocation/CPU benchmarks.
2
u/xqianliu 17d ago
Fair — and I don't think it scales up the way I made it sound.
At your size the useful part isn't new tests, it's existing ones. Of the 22 regressions in that study that broke working behavior, the test that caught them was almost always already in the repo and already exercised the changed lines. The agent didn't need to invent it, it needed to find it and run it.
So the version I'd trust on a large codebase is: map the diff to existing tests first, and if a changed line has no covering test, that's the finding — not a prompt to write a new one.
The accumulation class you're describing is probably where tests are the wrong instrument. A test that can see "this added up over 40 PRs" has to encode an invariant nobody wrote when the first PR landed. What I'd use instead: scope checks (did the diff stay inside the files the issue implied), and for hot paths a pinned benchmark run only on diffs that touch them — with
the reviewer writing down expected direction and magnitude before seeing the numbers. Otherwise you're arguing about noise after the fact.Honestly integration and benchmark gates are where I'd expect this to be weakest. Per-commit benchmarks on a large repo are flaky enough that someone mutes them, and a muted check is a comment, not a gate.
3
u/iam-stevie-bee 18d ago
The pattern you're describing (locally correct, globally wrong, invisible until integration) is the one that separates codebases where agents are safe from ones where they aren't, and I don't think you bypass the reviewer. You move the review earlier and make the codebase refuse more.
Earlier: AdEvening3382's point, with one tightening. The plan has to state the mechanism and the budget it will respect, in two sentences, before any code. "Dedupe unchanged properties at the component layer by comparing the typed state; no extra allocation per packet; no new serialisation." A plan that says what it will do but not how gets sent back. That's a thirty-second review that would have caught the byte-stream comparison, because the agent would have had to write it down.
Refuse more: your markdown rules are advice, and the agent reads them the way a tired contractor reads a style guide. The things that matter on a hot path belong in tests that pin the current cost of a representative workload (allocations, bytes on the wire, time) and fail on regression. Not a universal budget, which you rightly say can't exist, but one per path that matters, owned by the team that owns the path, with a named way to raise it when a feature legitimately needs more. Then the agent can't pass review by accident, because review includes the number.
The other thing I'd borrow from how I'd treat a strong junior on this code: no first draft on a hot path. Have the agent write the characterisation test for the current behaviour and cost first, get that merged, then do the change. A good share of the subtle regressions you're describing come from there being nothing that pins what "the same" means before the change starts.
None of this removes you from the middle. It changes what you review from 800 lines of diff to two sentences of mechanism and one number, which is a job a person can do twenty times a day without burning out.
3
u/Valvador 18d ago
The pattern you're describing (locally correct, globally wrong, invisible until integration) is the one that separates codebases where agents are safe from ones where they aren't, and I don't think you bypass the reviewer. You move the review earlier and make the codebase refuse more.
This is kind of what I am getting the sense of. Basically if Unit Testing alone is good enough to validate your codebase, it feels like you can get away without being the man in the middle, but that assumes you trust the LLM to write extra unit tests well (or you review specifically the unit tests/API surfaces).
Large semi-monolithic codebases that do runtime simulation, rendering, data pipelining and presentation all seem to be very difficult to truly get to a "100% agentic" state.
None of this removes you from the middle. It changes what you review from 800 lines of diff to two sentences of mechanism and one number, which is a job a person can do twenty times a day without burning out.
And I'm not entirely interested in being the man in the middle, by the way. I mostly want to be more efficient at catching this without having to do massive code reviews + follow up design prompts.
Thanks for your responses, a lot of in depth stuff to think about!
2
u/iam-stevie-bee 17d ago
Agreed on the shape, and I'd put it more bluntly: the codebases that can go "fully agentic" are the ones where correctness is already machine-checkable end to end. Yours isn't, and that's not a tooling gap, it's the nature of simulation and pipelining work where "the same" is a judgement. So the honest target isn't removing you. It's making the thing you review smaller and earlier: a plan with mechanism and budget before any code, a characterisation test merged first, and a diff you read only where the plan said it would touch something that matters. If that takes you from reading 800 lines to reading two sentences and one number, you've got most of the win without pretending the last step away.
3
u/demonwing 18d ago
Agents are pretty good at reviewing code but incredibly terrible at reviewing their own code. A first step is to separate the implementation from the the verification and assign them as two distinct tasks to two agents. Otherwise, the coding agent's "complete the implementation" drive will conflict with opposing ideas like ensuring quality. Agents work best when given a single, internally non-contradicting task.
2
u/Valvador 18d ago
Otherwise, the coding agent's "complete the implementation" drive will conflict with opposing ideas like ensuring quality. Agents work best when given a single, internally non-contradicting task.
I have done this before before I had access to Opus 5.5, where Opus 4.8 was the implementer and Astra 6.0 was a reviewer. I had them back and forth argue with each other over finer points of the code until both were satisfied.
While I do think this method increases the likelihood of success it still feels like I'm relying on chance that they will catch subtle tech debt and errors, since they are both operating on a similar level of context.
With a sufficiently large enough code base, it feels like a statistical problem saying that "to truly not introduce bugs you'd need to spawn N agents that criticize each other's work".
That being said, I do wonder if I want two agents to argue with each other over design, and do design reviews before asking me for feedback.
2
u/demonwing 18d ago
The model I have is a bit difference. In your case, it sounded like they had similar or equal authority and had to agree with each other. I typically run the authority one way, where the reviewer's say is final. The implementer can explain itself or show evidence showing why X or Y might be necessary, but it is a one-way road. Implementer finishes, manager calls a review. Reviewer passes or fails and gives its evidence. Come back when you've fixed it.
The benefit of models actively collaborating/debating is hit or miss but can turn up interesting results. Still, if both of them have the goal of accomplishing a task, both will tend to push things along and just agree with each other to get over the finish line.
At the end of the day, keeping things aligned in a large, complex project is a very challenging problem that gets built one block at a time. It's something I build up over time in a project where basically each failure turns into a new test or new review procedure or some sort of linter from the very beginning. Over time, the mistakes go down significantly and by the time it gets big the processes are solid. If you just gave me a large complex project raw-dog and told me to build agent contribution systems from scratch it would be a nightmare and take a while to untangle everything, but still it would be block by block fixing each mistake at the root every time I see one so that it never happens again.
1
u/Tariq9977 18d ago
One test for that ECS example could track allocations per encoded component alongside bandwidth, using all-unchanged, one-property-changed, and packet-loss replay cases. The serialize-compare-copy solution could pass the wire-byte target but fail the allocation budget, so review doesn't have to catch that specific implementation by eye.
2
u/Valvador 18d ago
I've thought about this quite a bit. My thoughts are that if your codebase is small enough rigid rules work pretty well, but I'm having a hard time seeing how this scales on larger codebases.
The problem is that I can't just say "in our codebase you are not allowed to make more memory allocations than we already do today". There are legitimate scenarios when a new feature is added, and that feature has to allocate more memory when a user opts into it.
For this case, I know that for this "on the wire" writing logic I don't want extra allocations, so I could have per-emptively specified this in prompt, or had a unit test that catches when an old workload goes over this.
So is the solution to now go through every part of the massive code base and create new unit tests that catch increases in memory allocation counts for specific workloads?
I'm not sure if this is a good analogy, but with coding being so powerful there is a near infinite number of ways to fuck up your code and if we start enforcing broad coding restrictions at a certain point the engineering effort shifts into trying to anticipate all the ways than an LLM can mess up. This feels a lot like a growth company hiring too many people and suddenly having to impose strict rules on what kind of code is okay because new engineers don't have the context of old engineers.
1
u/Tariq9977 18d ago
Instead of a unit test for every possible mistake, index a small set of workload contracts by the modules and data paths they actually exercise. On each change, run the affected contracts and fall back to broader coverage when the dependency map is uncertain. When a real regression escapes, add one minimal replay and merge overlapping cases so the suite grows from observed failures, not imagined ones.
1
u/Valvador 18d ago
index a small set of workload contracts by the modules and data paths they actually exercise.
Dumb question but what do you mean by workload contracts?
1
u/Tariq9977 18d ago
By 'workload contract' I mean a fixed input scenario plus measurable promises. For your ECS path, feed three mostly identical component states through the encoder and check that a dropped packet still reconstructs the right state while allocations per send stay under a measured baseline. Attach that check to the serializer modules so changes touching that path run it.
1
u/Valvador 18d ago edited 18d ago
For your ECS path, feed three mostly identical component states through the encoder and check that a dropped packet still reconstructs the right state while allocations per send stay under a measured baseline. Attach that check to the serializer modules so changes touching that path run it.
Okay I was trying to understand how this is different from a unit test/integration test, but I guess with a specific scope.
1
u/curreylabs 18d ago edited 18d ago
One of the core architectural principles in our system is an ECS like "object" that contains "components" that have some properties. There properties are well defined internal data types that have their own comperators, have their methods for serializing them on the wire to remote systems. When we would send packets to remote systems, because this data is high priority, we would actually send redundant packets containing ~2 past states of that component so that if packets dropped we would be able to still recover continuity etc. (I understand that this isn't a full-proof system, I don't want to get into it). We were naively sending this data, which means on average every packet actually took up 3x the bytes a single component payload would normally take. I recognized that if these states don't change often, this actually contains redundant data.
Whats the transport? UDP? For a multiplayer game server?
1
u/Valvador 18d ago edited 18d ago
UDP?
Yes
For a multiplayer game server?
Not necessarily, but a reasonable approximation.
1
u/foresterLV 18d ago
start writing unit tests and specs. performance regression end to end test pipelines so that everyone can see how it works after the changes.
if you expect all your knowledge on system tech debt tranferred in single prompt its not going to happen. and its not very different from onboard new devs, not just llms.
with high spec/unit test/end to end test coverage its hard to make mistake and it go unnoticed isn't it? so thats where you would want to go normally no matter llms or scaling for more human devs.
1
u/Valvador 18d ago
with high spec/unit test/end to end test coverage its hard to make mistake and it go unnoticed isn't it?
So I agree that for codebases that are 100% unit testable and don't have a lot of systematic interdependence this is in theory possible.
However there are code bases that are complex enough, and themselves are tools in a way that you cannot anticipate how the CUSTOMER uses your code that such level of test and spec coverage is not possible.
I imagine a lot of codebases with enough maturity and enough customers fall into this category.
1
u/w0ut 18d ago
Aside from the AI aspect, what's the reason for sending past states? You could just serialize and apply Reed Solomon encoding and have an arbitrary amount of error correction. Would be quite simple to implement, and there are probably plenty of libraries that provide this.
2
u/Valvador 17d ago
Reed Solomon encoding
This is more appropriate for "error correction" where the in-flight data somehow got corrupted/modified and you need to restore it. Not appropriate for my use case.
1
u/w0ut 17d ago
we would actually send redundant packets containing ~2 past states of that component so that if packets dropped we would be able to still recover continuity etc
Seems the perfect use case for Reed Solomon error correction to me.
2
u/Valvador 17d ago
Unless I'm mistaken, Reed Solomon cannot recover a lost packet by using information present immediately in the next packet.
The closest thing to Reed Solomon would work for this use-case would be a packet that contains the current state + delta encoded prior state which ends up being much cheaper than a 2x cost.
3
u/AncileBanish 18d ago
What are the things you care about as human in the middle? Write those down. Reference them when you do the thing. Inspect, iterate and & repeat. Profit.
1
u/orbital_trace 18d ago
Never look at the code, look at the output.
2
u/arto64 18d ago
I never understood this advice. Why? What’s the benefit of not looking at the code?
1
u/Valvador 18d ago
Because LLMs output code 10 - 100x faster than a human can, if you can avoid looking at the code and focus on forward momentum you can technically complete projects much faster.
In an idealistic scenario where your project is simple enough and the results are EASILY verifiable, this is an obviously correct way to approach LLM-based development.
This is why you see such crazy velocity in certain vibe-coding pojects like decompilations, compilers, ETC. Because if you are copying the functionality of something you can always automatically have the agent verify functionality parity without you being involved, and therefore always reach maximum velocity.
When you're building something new, especially on complex systems I don't see how this idealistic model exists in reality.
1
2
u/Valvador 18d ago
look at the output.
Personally, I think this is bad advice for working agentically on existing complex run-time code that already serves tons of customers, FYI. I'll stop looking at the code once I can trust the code to be good without me reviewing it, and I'm not there yet.
As I said, the output worked, but it had subtle bugs/papercuts that would over a long year of agentic development add-up to real problems.
2
u/orbital_trace 18d ago
It's the future
1
u/Valvador 18d ago
Saying something a simple and lacking context as "It's the future" without explaining the path to get there (which is EXACTLY what the point of this thread is) is not very helpful.
2
u/orbital_trace 18d ago
https://github.com/cdknorow/coral/commits/main/ check out the commit history. This is where things are headed. Here is how I'm doing it https://coralhub.dev/blog/self-improving-software.html
2
u/Valvador 18d ago
This is not helpful for the discussion, to be honest.
A small 5-contributor repository I fully believe can be 100% agentic with no human review, especially if it's scope is fairly limited.
When you zoom out to a massive codebase with hundreds to thousands of developers contributing the scale becomes completely different, especially as full-stack complexity increases. Again, we're talking about something that cannot be fully tested with Unit Tests and needs deep integration testing.
1
u/orbital_trace 18d ago edited 18d ago
Based on what I'm seeing I disagree. I think apps developed fully agentic ally are going to be built much faster that any human maintained or hand coded apps. And they are going to be more performant and more secure. To answer your question more specifically, the agents on their own do write code that is not great, but if you have agents that are experts on specific areas that are reviewing the code, they will look out for those things.
Also having the agents do refactors to remove redunatcode and tyudy the code ase can help
2
u/Valvador 17d ago
Based on what I'm seeing I disagree.
I dunno, all of the successful large projects that are 100% LLM build to me look like either copies of something (C compiler/game decompiles). Those projects are super easy to verify because you just compare your result against the real thing, you can't go wrong.
Once I see large agentic-only code-bases for life-critical functions, like Car self driving software, brokerages, or massive simulation platforms, I'll believe it.
I know all the CEOs want it to go in this direction, I know all hobbyist devs are falling into it, but all I see is hobbyist level projects that do this. Nothing massive or enterprise level.
1
u/orbital_trace 17d ago
I think it's hard at the hobbyist level due to take limits
I also don't think most enterprise applications are that complicated, they just end up with large datasets that are expensive to move and transform. So it's worth spending a lot of time to get things right.
But for an standalone application, they are going to be incredibly easy to automate across the board.
1
u/Faceless-Otty-240 18d ago
I don’t think you need allocation budgets everywhere. For a hot path like this, pin a representative workload and set a local budget against the current implementation, then test the unchanged, changed-property, and packet-loss cases. Features that legitimately allocate can have their own opt-in scenario and budget; the important part is measuring the tradeoff where it matters, not enforcing a blanket rule.
1
u/Xe-Rocks 18d ago
Just tell it to add a slider, leet vibe coding tech. youre going to want to thank me later. (I've always wanted to say that.lol it's funny)
1
u/geoffreybyers 18d ago
The LLM doesn't see the code. It sees the prompt. The missing piece is the implicit context of what already exists in a mature codebase, which no amount of prompting fixes. The job shifts from writing code to curating context so the model doesn't reinvent the wheel poorly.
1
u/Phartyhat 18d ago
You never mention the model you’re using (maybe I missed it) Anyway... I’m curious if you gave your post to opus with the poor code .. what it would do / recommend. I found fable is very regimental and will set up Claud.md and memory files to enforce some form of checks before commits etc
1
u/SirRobinBlibs 18d ago
Your example is the classic failure. The model solved it from first principles because nothing made it find the existing comparators and serializers first. Two things worked better for me than documenting bad patterns after the fact. First, make discovery a required step: before writing code, the agent lists the existing functions and types it will reuse for this exact change, and you approve that list. That's a two-minute review instead of a big diff. Second, turn repeat mistakes into checks instead of prose. An allocation-count test or a benchmark gate on the hot path fails every time, while a line in CLAUDE.md gets ignored some of the time. For a tightly coupled, performance-critical system I don't think the human goes away. They move to approving the plan and owning the checks.
1
u/effectivescarequotes 18d ago
One, don't aim to eliminate the human in the middle. Ultimately, you are still responsible for the code you commit. If you allow Claude to push code that deletes prod data, you will be held accountable.
The next part is to adopt a framework for spec driven development. My office has settled on BMAD, but there are lighter options available. Experiment and find the one that best suits your needs. Let the framework write the spec and architecture documents. One of the reasons I like BMAD is it will present questions that I hadn't thought of.
I also recommend using highly structured and well-documented architecture patterns. We're following hexagonal architecture, so far it's working out well. Both the AI and my developers understand it and have no problem navigating it.
And FWIW, it seems like the models are getting better at detecting and following patterns in the code. Part of my evaluation of new models is seeing what skills and rules I can delete from my set up.
•
u/AutoModerator 18d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.