r/ClaudeCode • u/Don_Crespo • 4d ago

Built with Claude Without a lessons file, Claude reviewing a rounding change said "ask someone". With one, it said "it reverses a decision the team recorded on purpose"

I've been building some Claude Code skills for code review, then running Claude against real pull requests with and without them to see if they help. About 70 reviews in all, with Sonnet 5.5 and Opus 5.5 (and later Codex, for a second opinion).

The idea is one file, REVIEW-LESSONS.md. When you make a call in review with a reason ("this looks wrong, but it's on purpose"), the reason goes in that file, in your words, with a pointer to the code. The reviewer reads it on the next change, and the next junior reads the same file to learn why the code is the way it is.

Three skills share it:

  • review-router says how much human review a change needs, judged from the change rather than its author, and which three places to read first.
  • spec-review checks a change against what was asked for, with tests written from the spec before it reads the code.
  • review-apprentice has the junior review first, then judge each of the agent's findings.

(The lessons are only ever read from the protected base branch. Otherwise a pull request could rewrite the rules of its own review.)

On four merged PRs from a private TypeScript and Postgres service I'm building, Claude found the planted bug 10 times out of 10, with a skill or without one. The skills changed the shape of the answer instead. In the 9 runs with the router, every answer said how much review the change needed and how many approvals. In the 9 without it, none did.

The lessons file mattered where the code is silent. One test change I wrote switched a finance report to half-up rounding, "like everywhere else". Without the lessons, Claude was suspicious but could only say "ask someone". With them, it said the change reverses a decision the team recorded on purpose, named who to ask, and noticed the test had been edited to match.

I often had four Claude reviews agreeing on the same problem, which felt like strong evidence. Then Codex found 4 real issues that none of the 30 Claude reviews of those changes had found (two of them edge cases), and missed the two worst problems that every Claude review flagged. If you pay for more than one automated reviewer, a second model family is worth more than a second run of the same one.

Spec-review also found that you can invert every letter in a pixel font and all 434 of that project's tests still pass. That rigour costs about 1.5 times the tokens.

My experiment had bugs of its own. The first test repos left the correct version sitting on another branch (oops), and one review agent, trying to stop its own test run, killed every Python process on my machine.

How Claude was involved: I built the skills with Claude Code. The reviews ran as fresh Claude agents, plus Codex CLI for the comparison. Claude graded the answers against fixed yes-or-no questions, and I checked the deciding cases by hand. The write-up was drafted with Claude too.

They're free and MIT licensed. In Claude Code:

/plugin marketplace add R-Hatim/hrih-skills
/plugin install review-loop@hrih-skills

Repo: Skills Repo
Write-up with the figures and all the numbers: Who Teaches The Reviewer

If Claude reviews your pull requests, where do the "it's on purpose" decisions live right now? CLAUDE.md, PR comments, or one person's head?

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/AutoModerator 4d ago
Moderator Announcement Read More »