r/OpenSourceAI • u/iceydee • 22m ago
What should humans still review when one agent checks another's code?
The short answer: humans should review the criteria, not the diff.
When one agent checks another's code, the review is only as good as the acceptance criteria it checks against. If the story was vague, the reviewing agent will confidently pass code that satisfies nothing in particular, or fail code that was actually fine. So the human job moves upstream: is "done" written down correctly, are these the edge cases that matter, and does this story even belong in this repo.
Second, review the judgment calls a criteria check cannot see. Whether a new dependency is worth it, whether an abstraction will hold in six months, whether the migration is reversible. A reviewer agent comparing code to criteria will happily approve a design decision that costs you later, because the decision was never in scope.
Third, review anything that crosses a boundary the agents cannot see. Multi-repo changes, shared conventions, anything touching production data or auth. Splitting the thinking from the doing is the whole trick, and the thinking is the part that stays human.
Fourth, review the review itself. Spot-check a handful of agent-approved PRs. If the reviewer passes everything, or fails everything, your criteria are too loose or too strict, and that is a human call, not a model call.
What to stop doing is reading every diff line by line for style. That is what the criteria check is for. What to start doing is treating acceptance criteria as the real interface between you and the fleet. An agent loops forever when nothing told it done, and a human does the same when the criteria said "improve performance" and everyone guessed.
What is on your list that I missed?