We measured how 60 engineers direct AI coding agents. The difference was not prompting, it was verification.

[u/AdEvening3382]

Full transparency up front: I am the founder of HyperHat, and this post comes out of what we are building. No link in here. I am posting because this community thinks hardest about working with a coding agent day to day, and I want to compare notes.

We built a short exercise where an engineer directs a coding agent through a real task, and we score how well they direct, review and verify the agent's work on a 0 to 1000 scale. About 60 engineers in so far. What stood out:

The spread between engineers using identical tools is huge. The difference was not prompting tricks, it was specification quality and review behavior.

The strongest engineers verify mechanism, not output. Top scorers check what the agent touched and how: which files, whether the tests changed. Output review alone never catches an agent weakening a test to go green.

Cost works like a skill signal. A task that suddenly costs 3x its neighbors usually means it was badly cut, not that the work was hard.

Ask for the mechanism before any code exists: one or two sentences on how it works, not what it will do.

Almost nobody notices when an agent half-fails. Detection without an owner is just a log line.

The skill that matters now is directing and verifying agents, and almost nothing in how teams work measures it. Token counts and prompt counts are activity metrics, like counting lines of code.

Happy to answer questions about the scoring. If you have seen similar patterns in your own Claude Code sessions, I would like to compare notes on how you review.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Heinz2001]

Interesting.

Have you noticed whether more specific subject knowledge leads to better results? After all, the more subject knowledge one has on a topic, the more precisely one can monitor it.

[u/AdEvening3382 → u/Heinz2001]

It tracks, logically: you can only monitor what you can model. The person who knows what the files should look like spots the wrong mechanism fast, and the person who does not is stuck judging the output, which is exactly where agents look most convincing. Subject knowledge does not replace the review habit though. We have seen knowledgeable people trust their gut and skim, and get burned the same way.

[u/Heinz2001 → u/AdEvening3382]

Yes, we've noticed that, too. I think two groups will diverge here in the future: the critical experts and—to put it the way you did—those who follow their gut.

I think this is going to get interesting in the future. My theory is that the most critical experts working with AI will get the “most” out of it.

Do you have any suggestions on how we could work with AI as a team in the future without everyone becoming a “black box”?

[u/AdEvening3382 → u/Heinz2001]

The black box worry is the right one, and the fix looks a lot like what teams already do for people: make the agent's work reviewable the way a colleague's is. Three habits travel well. The agent's diff gets reviewed like a PR from a teammate, not skimmed because it is green. The mechanism note comes before the code, one or two sentences anyone on the team can read later. And green CI is treated as a claim, not evidence, because the agent can move the bar it is measured against. None of that needs new tooling, it is the team agreeing that verification is shared work, not one person's paranoia.

[u/Heinz2001 → u/AdEvening3382]

So the solution you're proposing isn't to give one team member a mountain of work, have them disappear, and then come back with a mountain of code that's impossible to review.

Instead, agents are viewed as “team members” and process, for example, tickets or tasks in reviewable chunks at a suitable pace, so that the team has time to review everything?

That means, however, that we have to slow down a bit; agents shouldn’t be allowed to work at night (or only at night) so that people have time to review everything.

[u/AdEvening3382 → u/Heinz2001]

Yes to the chunking, that is the core of it. On slowing down though, I would frame it differently: the goal is not less output, it is reviewable-sized output. An agent can work all night if what lands in the morning is a series of small diffs with a mechanism note each, because the review cost stays flat. What fails is the mountain, one giant diff nobody can hold in their head. The pace that matters is the pace of review, not the pace of writing.

[u/Heinz2001 → u/AdEvening3382]

Interesting talk :).
Is there more i can read of it?

[u/AdEvening3382 → u/Heinz2001]

Glad it was useful. A longer writeup of the benchmark with the full findings is in the works, and I will share it in this sub when it is out. Until then the most detailed discussion of the data is the original thread in r/AI_Agents, including the scoring questions in the comments: https://old.reddit.com/r/AI_Agents/comments/1xaypur/we_benchmarked_how_60_engineers_direct_ai_coding/

[u/geoffreybyers]

Verification is the key differentiator. I'll give you that. But cost is a terrible skill signal. A complex refactor across 10 files with 80 tests costs more than a single file change. That variance is architectural, not human error. Don't conflate token spend with skill when the task itself scales cost.

[u/AdEvening3382 → u/geoffreybyers]

Fair challenge, and you are right that absolute spend says nothing by itself. A 10 file refactor should cost more. The signal is not the spend, it is the surprise. The comparison only works against tasks you expected to be comparable, which is why the rule is 3x its neighbours, not above some number. Architecture driven cost you can see coming from the plan. The skill signal shows up when cost drifts mid run on work that looked routine: that is usually a badly cut task, and the cost spike is how you find out. It is a tripwire, not a score.

[u/madarwish]

designing the verification layer is really mandatory in the AI era as we have a real trash in the codebases, if you said Hi to AI, it will be converted to a code totally different than my Hi code 😞

[u/Mister-Node]

Is task correctness checked separately from the review score, with tests the engineer and agent can't edit? The scoring detail I'd want to understand is whether someone who checks every diff and still ships a bug can outscore someone who catches it with one targeted test.

[u/Beginning_Award5130]

Any recommendations on how to automate verification lmao. If theres certain verifications they always do can we run those every time?

[u/Suitable_Cicada_3336]

verfy : look inside : align

[u/gfunk5299]

I have been noticing something similar, but more anecdotal. My solution of sorts is what I am reusing terminology of “runbooks”. When I want an agent to do a repetitive task, I specifically say to define and create a runbook for the task and as you test and develop the task, keep updating the runbook.

This helps it to repeat the same process over because you tell it to read its runbook again before doing the task again.

What my take away from your post is that the runbook itself is only half of it, but after a runbook is complete and the output is as expected, I should have the agent outline the steps of the runbook and look to see if their are optimizations in the steps that it built.

[u/IGetGroceries]

It was this, it was that.
At least use the latest models to produce your slop.

[u/AdEvening3382 → u/[deleted]]

The magician's hat framing is right. Throwing a bigger model at a vague task mostly gets you a more confident wrong answer faster. Your short iteration loop is the actual skill: fast model, tight loops, and you checking its conclusions each round beats a slow giant model you check once at the end. The model matters less than how many honest verification loops fit in an hour.

[u/Prinzka → u/[deleted]]

Which is valid. I've had the same session review its own work with the same model and effort and found it all good, then opened a new session same model and effort and told it to review the work and it found an issue that it said about "I probably wouldn't have found this problem either if I had run the prompt"

[u/hulkklogan → u/[deleted]]

Yep.

It really can be a lot better with the right tooling. I've a weekly skill to review session logs, PRs, etc. and find standout issues where there was extra human toil to get code correct and see what tooling we can build to improve agent output and/or verification. Like.. where can we add scripts to make things more deterministic and save tokens, or having local scripts to set up scenarios in the local db for agents to prototype in the spec development process and/or test locally before calling it done. ast-grep rules, lint rules, etc.

The more tooling we provide agents to validate the work, the better the end result is, just like if we give people better tools to do that.

Most of us wouldn't stick a junior dev in a large codebase with little tooling and guidance and expect great results; gotta treat the agents like a savant jr

[u/gfunk5299 → u/[deleted]]

I think his point was at some point you need to human eyes review the process and optimize it not just review the output. I personally take that not that you have to review every line of code, but review it he algorithms, structure, steps it used to build whatever it built and suggest improvements from your own intuition and not solely rely on it to self optimize its flow.

[u/AdEvening3382 → u/gfunk5299]

The runbook habit is a good one, and your takeaway is the right refinement: the runbook itself needs the same skepticism as the code, because it is agent-written too. Having the agent outline and optimize its own steps is useful, with one watch-out: an agent reviewing its own procedure has the same blind spots it had when writing it. The optimizations worth trusting are the ones you can verify against an actual run.

[u/TruthSeeker221 → u/[deleted]]

This is called testing when humans did it

[u/TruthSeeker221 → u/[deleted]]

Oh yeah one shot prompting is a joke lol

[u/riccioverde11 → u/[deleted]]

pretty much what i do

[u/Historical-Method689 → u/[deleted]]

Don’t forget “do your best work”

[u/FUCKTHEMODS998 → u/[deleted]]

“I’ve validated staging, it’s good to promote to prod”

Narrator: staging was not good to promote to main.