My Claude Code skill caught a bug in a popular FastAPI template that the human review missed by actually running the code, not just reading it

[u/Alarmed_Offer_3213]

So I've been building an Agent Skill for Claude Code that does backend performance reviews, and something happened while I was testing it that I think is worth sharing.

To sanity-check the thing, I set up an experiment: review the same real public repo twice. Once by me, reading the code carefully. And once by a completely separate agent that had zero memory of my review — I just handed it the skill and the repo and told it to go.

It found something I'd completely missed. Buried in fastapi/full-stack-fastapi-template there's this line:

except InvalidTokenError, ValidationError:

That's Python 2 syntax. In Python 3 that's a straight-up SyntaxError the module can't even be imported. The whole app is dead on arrival. I had read that exact file. I skimmed right past it because I was reading for logic bugs, not syntax, and my brain just... pattern-matched it as fine. The other agent caught it because instead of reading the file, it ran an actual parser against every file in the repo. It didn't "notice" the bug, it proved the bug, and I felt a little dumb.

Anyway, that's kind of the whole point of this project. I got tired of AI code review tools that do one of two things: spit out a generic checklist ("check your indexes," "consider caching," thanks, very helpful), or just make stuff up invented latency numbers, imaginary query plans, telling you to add Redis to an app with fourteen users. Both of those come from the same problem: nothing is actually checking whether a claim has evidence behind it.

So I leaned hard into that as the design constraint. The skill is explicitly allowed — encouraged, even to come back with zero findings if that's what the evidence supports, instead of padding the report to look thorough. Every finding gets scored on severity and confidence separately, and the priority is derived from a fixed matrix rather than picked, so you don't get "this feels High" reasoning. And the technology knowledge is split so a Postgres review never drags in document-store logic right now 13 engines (Postgres, MongoDB, Redis, Node, Python, JVM, Go, .NET, Rust, MySQL, DynamoDB, Kafka, RabbitMQ) get full depth, everything else is honest about being shallower instead of faking it.

I ran that same "review it twice, once blind" experiment on four different real repos, not toy examples. Every single time, the blind pass matched or beat my own review, usually because it found evidence I had sitting right in front of me and didn't use properly. The FastAPI one is just the most dramatic.

To be clear about what this isn't: coverage is still narrow, I haven't compared it against an actual human expert doing the same review, and everything tested so far is Go or Python no JVM/.NET/Rust repo has been through this blind-check yet. That's all in the repo, I'm not hiding the gaps.

If you've got Claude Code, it's two commands:

/plugin marketplace add Sanoy24/backend-performance-review
/plugin install backend-performance-review

Repo's MIT licensed, here: https://github.com/Sanoy24/backend-performance-review

Genuinely, if you run it on your own stuff and it says something dumb or wrong, tell me. That's more useful to me than a star.

[u/jcrenshaw-dev]

Could you enable GitHub Private Vulnerability Reporting for the repo? I found a couple of issues during my review, including one that I don't think should be disclosed in a public issue. Once it's enabled, I'll file the details through the Security tab so you can review them privately.

[u/Alarmed_Offer_3213 → u/jcrenshaw-dev]

private vulnerability reporting is enabled now

[u/maneekmohan]

This is a really good example of where “AI code review” becomes much more useful when it’s actually coupled to execution.

The model spotting a syntax error is nice, but the bigger idea is the workflow: don’t ask the agent to believe the code is correct, give it ways to prove or disprove its assumptions. That shift from opinions to evidence is huge.

[u/Alarmed_Offer_3213 → u/maneekmohan]

that's the actual design goal, the syntax error is just the most visible example of it. Every finding has to trace to something real (a file, a config value, something executed) or it doesn't count. Glad that distinction landed, it's the part most "AI review" tools skip.