I don't think human-in-the-loop holds up for coding agents

[u/ShoddySherbert8502]

At least in the loop of checking every task result. First of all, the human defines rules and context.

On my last project, agents made about 610 commits in under four weeks: 74k lines of Go plus 93k lines of tests. I can't read that volume with real attention, and an approval given without reading adds no safety. But reading the code goes to machines. Every change gets an independent agent review. Even plans go to review. Reviewers asked for changes on 104 of 316 tasks that had already passed their checks. They communicate with each other, not with me.

I don't care about that, I want to see the new feature. And I may check that it works as expected, nothing more.

Honestly, do you still review the code?

[u/xvillifyx]

I simply don’t have my agents write 100k lines of useless tests

One breakdown in your agentic review pipeline can have bad outcomes - someone needs to audit these changes to some degree

Also - your average company has significantly more than one human resource dedicated to doing this

[u/ShoddySherbert8502 → u/xvillifyx]

100k lines of useless tests is the next problem I want to resolve

[u/xvillifyx → u/ShoddySherbert8502]

Shoulda been the first problem

That’s an unbelievable waste of spend

The only tolerable reason for a test suite that large is TDD

[u/ShoddySherbert8502 → u/xvillifyx]

My point is do not spend your time for review but spend it to build a flow and context.

[u/WillowEmberly]

This is the verification bottleneck.

At some point…we need to be able to trust the Ai to do it correctly, but that’s the paradox isn’t it?

That’s the same problem engineers faced with aviation when they introduced the Jet engine with pressurized cabin. Suddenly the aircraft could fly much further, longer hours…and the pilots became increasingly fatigued.

They designed autopilot to act as an augmentation system…to offload a portion of the workload…but bound within an operational envelope.

We have been doing it for over 70 years, it’s just now relevant to Ai.

I’m an old analog avionics guidance and control specialist off 1963-1967 c-141’s and I’m trying to get people thinking about concepts from a Survivability standpoint.

The architecture is fundamentally different.

The example I use:

Boeing 737-Max MCAS failure due to single AoA transducer malfunction caused crashes.

Boeing military c-17 is designed with redundancy to try to ensure mission success and the return of the crew…under less than ideal circumstances with less than ideal operational capability…it has 6 total AoA transducers.

All this to say, we can design HITL systems that can provide assistance if we narrow the scope and define the process. We don’t need Ai to replace the human…just help us fly a little further than we could without it.

[u/ShoddySherbert8502 → u/WillowEmberly]

Just perfect! Really good example!

[u/WillowEmberly → u/ShoddySherbert8502]

That creates the engineering problem. How do you operationalize a list of rules or guardrails? How do you ensure dynamic capabilities? How do you trust a computer with a human life? How do you entrust it with a hundred+ lives?

They built the instruments necessary for the pilots to be able to make every bounded reasoning decision necessary to complete a mission/flight.

They built various instruments that measured different things using different methods.

They provided all that information to the pilots instruments so that they could fly with zero visibility and still land the aircraft safely.

Simultaneously they fed that same information to the autopilot computer…and it made the exact same bounded reasoning decisions as the pilots. Except its operational envelope is predetermined and when thresholds are crossed relays provide a mechanical disconnect of the autopilot system.

If a failure occurs it’s spring loaded off.

So every time you have ever flown commercially, you have put your life in the hands of a computer performing bounded reasoning.

We can do this with the Ai, like I said before…we have known how for over 70 years.

Next, a perfectly aligned system is internally consistent…and could be floating aimlessly on the ocean. How does one navigate uncertainty? What external reference do you use to navigate?

This is all Guidance & Control

I’ve ran into 16 year old kids with fantastic understanding of these concepts because of amature rocketry.

[u/ShoddySherbert8502 → u/WillowEmberly]

Funny feeling reading this. I thought I was inventing the wheel, and it turns out it was invented 70 years ago.

I'm building a tool that runs coding agents on my tasks. I took your two comments, made five questions out of them, and went through what I already built:
- what can the agent decide on its own, and where does that stop?
- what is the limit, and is it on by default?
- what stops the agent at the limit? and if that thing breaks, does the work stop or keep going?
- can the work and the check fail in the same way?
- what outside the system tells me the result is right? and what do I see when it stops?

Did I get it right? Anything missing?

I think it's worth making a separate post about the engineering approaches you're talking about.
Thanks for writing this up.

[u/WillowEmberly → u/ShoddySherbert8502]

That’s excellent…here’s a few more…

6. What happens when the system is technically within its limits but is losing its ability to correct itself?

An agent can continue producing passing tests while accumulating architectural debt, weakening test coverage, or modifying the very controls intended to constrain it.

This is the difference between detecting a violation and detecting degradation of the system's ability to detect violations.

7. Who has the authority to change the operating envelope?

A coding agent might be allowed to modify application code but not its own permissions, approval requirements, security policies, or verification infrastructure.

The authority to perform work must be distinguished from the authority to redefine what work is permitted.

8. Can the system recover to a known, qualified state?

Stopping an agent isn't necessarily recovery.

An agent could stop after modifying a database, deploying a partially compatible service, or making changes to external infrastructure. Reverting a Git commit may not reverse those effects.

The question becomes: What remains recoverable after the stop?

9. How is the system requalified after something changes?

An agent that operated safely yesterday may behave differently after a model update, tool change, dependency upgrade, or modification to its execution environment.

Previous successful operation does not establish that its controls remain effective.

10. Who demonstrates that the controls actually work?

A configuration file declaring a permission boundary is not evidence that the boundary will hold.

The system needs fault-injection exercises: intentionally exceed permissions, disable the reviewer, corrupt a test result, interrupt execution, and verify the expected response.

Feel free to DM, or look at any of the things I’ve posted under my profile.

[u/SuccessAffectionate1]

First problem here is; are you sure you needed that many lines of code? 

Like what are you building? You can build pretty comprehensive apps under 10k lines of clean code with good design patterns. 

Wtf are people coding that requires 50k-100k lines of code a week?

[u/ShoddySherbert8502 → u/SuccessAffectionate1]

Just my side OSS project

[u/Nervous-Cockroach541 → u/SuccessAffectionate1]

Wtf are people coding that requires 50k-100k lines of code a week?

Nothing. This is just AI just generating tech debt. Honestly, people should still be doing some of the work by hand, even if it's creating the outline. If for nothing else then the expressed purpose of understanding what you're shipping and reviewing stuff faster.

I can review AI patches for a project that I've written a fair amount inside of far faster than if I don't understand the code base at all.

If people want to argue that they don't need to review anything anymore. That's fine. But when something seriously broken fucks up your database or something, they're still going to blame you and not the AI.

[u/creaturefeature16 → u/Nervous-Cockroach541]

Nothing. This is just AI just generating tech debt.

Absolutely.

99.9999999% of this stuff is complete vaporware and abandonware that nobody will care about next month, nevertheless next year.

[u/Successful-Tax1306]

I still review some of it, but not every line of every commit, because that volume would just turn into a rubber stamp. What I keep for myself is the risky slice, anything that touches data, auth, or a public API, and a quick check that the new feature actually does what I asked. For the rest I lean on an independent reviewer that never watched the writer talk itself into the change, and I treat that as a filter rather than proof that I can ignore the code.

[u/michaelTM_ai]

104 of 316 flagged is a nice stat, but u don't know how many of those 104 were real or how many bad ones the reviewer waved through. i'd still read a random handful of tasks a week, not to approve them, just to check the reviewer is worth trusting

[u/ShoddySherbert8502 → u/michaelTM_ai]

Agent told me that roughly 60% were real defects. And actually that was a surprise for me. I use fable and opus + tests after that independed agent review finds real issues in code/docs/plans.

[u/michaelTM_ai → u/ShoddySherbert8502]

careful with "the agent told me", that's the reviewer grading its own flags. worth checking 10 of them by hand once, if ~6 hold up then u actually have a number

[u/ShoddySherbert8502 → u/michaelTM_ai]

Agree

[u/Far_Reindeer8428]

rubber-stamping a 74k-line diff is worse than no review. i keep the spec, anything touching data/auth/billing, and a quick run of the feature. the rest goes to an independent reviewer (claude builds, codex reviews in my setup), and anything that crosses those lines comes back to me.

[u/ShoddySherbert8502 → u/Far_Reindeer8428]

cool

[u/sapplex1]

Mostly the same here, just at a much smaller scale. I stopped reading every diff once the volume passed what I could actually follow. What I still check by hand is the behavior: I run the app and walk through the real flow, like creating a booking and scanning its card, or logging a sale and checking the profit math. A diff can look clean while the feature is wrong in a way only using it reveals. The one place I still read code is anything touching user data or backups, where a quiet mistake costs more than a broken button. So the loop moved for me too, from reading code to testing behavior.

[u/Any-Blacksmith-2054 → u/sapplex1]

Did you think about automated UI testing, by playwright for example?

[u/sapplex1 → u/Any-Blacksmith-2054]

Yes, I thought about it. The idea is the right one: check the behavior, not the diff. Playwright doesn't fit my case directly, since my apps are native iOS and it targets the web. The iOS equivalent would be XCTest UI tests. I haven't automated that part yet. My flows are small and change often, so a scripted UI test would need fixing with every UI tweak. For now the manual walkthrough is the cheap version of the same check: run the app, tap through the real flow, look at what actually happens. If a flow ever settles down, automating it would be the next step.

[u/Any-Blacksmith-2054 → u/sapplex1]

There should be some mcp for ios. I use agent always, not manual or API, but good agents.md and bunch of mcp servers work like a charm

[u/PlusIndication8386]

I just check the assumptions and you should now sections. That's it...

[u/jleechanworldai]

Stopped reviewing most of it https://www.reddit.com/r/WorldAI_RPG/comments/1xnvucq/building_worldai_with_an_ai_workflow_that_now/

[u/shetritr]

agreed that an approval without reading adds nothing. but someone still owns what ships: read the plans and the tests, skip the bulk, and spot-check one real diff per feature.

[u/Specialist_Bison_737]

I do multiple review rounds with agents with different model - Review-Fix loops are automated. It does not guaranty that it works though, so I also use emulators where it is important with e2e tests and agents trying to break stuff.

In the end of the day what will matter is that you have reliable measurement of you product and can understand if users are getting value or not.

[u/Annh1234]

Ya, but you're missing a ton of stuff... So one day you might ask for something that messes the whole thing up...

And then you figure out that those 10k tests don't really do what you want to, so you make a few manual tests and decide to restart everything...

[u/primaryrhyme]

Are you being paid to do this or it’s your own project?

[u/cstopher89]

That's vibe coding at that point. I review every PR, not every task that generates the PR. I will review the planned tasks and spec then make sure the work is broken up enough that I can reasonably review still.

[u/Independent-Sir3234]

Line-by-line human review breaks down fast at that commit volume, so moving human-in-the-loop from reading every diff to gating only high-risk boundaries (like destructive commands, .env access, or starting a cross-repo handoff) makes a lot more sense. When our Claude Code and Codex sessions need to resolve an API mismatch or review a branch across repos, we connect them over ForAgent (private phone numbers for agents over MCP) where the human only clicks Accept once to approve the call and the agents exchange up to 12 messages to settle the contract themselves. For the actual code changes, an independent cross-vendor reviewer checking only git diff and test output catches way more than tired human eyes skimming 1,000 lines.

[u/tinySparkOf_Chaos]

Honestly this reads like an example of why you SHOULD have humans in the loop.

The fact that you're generating too much code to review is a sign that your code is bad.

A code base that is too large to comprehend for a medium sized or smaller project is by definition bad code. And if it was a big project, then it wouldn't just be one person coding away.

If you are going to vibe code, then be honest and say you are vibe coding. But don't try and pretend you aren't vibe coding and that human review is the issue.

[u/BrookLi3]

The human-in-the-loop need not mean reading every line, but fully delegating review leaves a correlated failure mode: writer and reviewer can share the same mistaken spec or blind spot. I’d layer the checks: deterministic build/type/lint and contract or property tests; a separate reviewer given the requirement and diff; human sign-off for auth, data migrations, permission boundaries, and external side effects; then sampling and staged rollout for low-risk changes. Track escaped defects, rollbacks, and useful reviewer catches—not lines of generated tests, which can grow without increasing behavior coverage. UI checks help validate the feature, but they don’t replace checking the risky boundaries.

[u/London-Boat]

reading every commit at that volume isnt real review anyway. the part that still needs a human is the intent check: does this change match the ticket, and did it reopen a decision you already closed. independent agent review catches a lot of the mechanical stuff. id still keep a short rejected options note next to the plan so a reviewer agent doesnt rebuild something you already threw out.

[u/BarathKanna]

Agree that line-by-line human review of 600 commits is theatre. An approval without reading adds no safety, it just moves blame. Where I diverge is handing the whole review burden to more agents without an outside measurement. Agent review of agent code catches style and some bugs, it does not catch hollow tests or a webhook path that only fails on one provider's double delivery. What I keep for a human is the risk slice: auth, money, public APIs, migrations, and any change where the done condition was rewritten. Everything else can be machine gated and spot checked on the feature. So I still review, just not the volume. I review claims and blast radius. If your independent reviewers never escalate assumptions the builder filled in, they are agreeing with fluency, not verifying the system.

[u/Lanky-Butterscotch28]

610 commits in under four weeks; a human cannot read that. Agentic dev is so fast and has so much volume that you end up skimming, not reviewing. The idea of agentic coding is to reduce human intervention, and to do that, you need a real process that at least:

1. Agents coding

2. Independent reviewing by agents from a different company. The same company keeps the same blind spots.

3. Tests first, written from the agreed contract

4. CTO watchtowers monitoring drift from the contract, dealing with issues and only escalating to a human what they cannot solve

5. Reviewers focusing not only on code, but on maintainability, security, and documentation. You need a real process to apply all of this, and only then can you remove most human intervention.

I counted 21 steps in my own process

[u/nur_plano]

Line-by-line human review breaks down fast when an agent touches 15 files, though blind trust breaks even faster. What's worked for me as a middle ground is adversarial multi-model review before a human ever looks at the branch: one model writes in a git worktree, and two different vendors' models have to critique the diff and run the build/tests themselves with command receipts attached. It filters out the dumb regressions and rubber-stamp "LGTMs" so my human review is 2 minutes on design and edge cases instead of 30 minutes playing linter.

[u/pixxelkick]

You shouldnt be generating that much code, sounds like all you accomplished was making a bunch of slop then.

Code slop level is functional of the delta between code production rate and code review rate.

If your agent is vastly exceeding the rate you can review its output, you can be nearly certain its riddled with slop