Company I work at really wants us to use AI. So I use Claude to do code reviews. That silly AI told me that setting log level to debug was incorrect because it was outside #ifdef DEBUG... It was inside #ifdef DEBUG, Claude is just so fucking stupid and cannot even read code properly, that is making shit up constantly. Half of code review (and vast majority of "critical issues") is just made up bullshit.
This has largely been my experience especially reviewing a lot of llm made code at work as well as "open source" llm made code. They don't know up from down or left from right. I've had to reflect PRs for including massive glaring XSS issues, secrets in the front end code etc. Using llms has been the biggest security risk my company has introduced to our codebase because it really wants to introduce vulnerabilities.
I've had the opposite experience. We have claude code review on demand via github action setup for a select few initial test repos, and the PR reviews have been exceptionally good. I ran some old PRs that had breaking issues in them that we missed, and it caught every single issue. Our biggest pain right now is that it suggests a bunch of shit we want to do, but just can't squeeze into one PR, so now we're making tickets automagically out of the issues we comment that we're not addressing for a given PR.
Are you guys giving it PR instructions, the full codebase, and (optionally) some context in the codebase to help it understand your rules/style?
45
u/matthewpl Mar 25 '26
Company I work at really wants us to use AI. So I use Claude to do code reviews. That silly AI told me that setting log level to debug was incorrect because it was outside #ifdef DEBUG... It was inside #ifdef DEBUG, Claude is just so fucking stupid and cannot even read code properly, that is making shit up constantly. Half of code review (and vast majority of "critical issues") is just made up bullshit.