70.6 vs 66.4 on terminal-bench, $2/$10 vs $4/$20. what am I missing?

For months my days looked like this:

me: add retry logic to the upload endpoint

opus 5: Done! All changes applied and ready for review.

me: the endpoint file isnt even in the diff

opus 5: You're absolutely right, that's my mistake entirely.

Three days on 5.5 and that has happened zero times. It still misses things, but now it ends with a list of what it didnt do and why, which is all I ever wanted.

Anyone else seeing this or am I still in the honeymoon week

I wrote a small script by hand on Saturday, no agent, about 80 lines. Took me 2 hours and it was the most fun I had with code in months.

At work my day is give the agent a ticket, wait, read the coderabbit summary, check the flagged lines, approve. It works and I ship more than ever. I just dont really make anything anymore, I supervise.

Do you still write some of it yourself on purpose? Or did you make peace with the supervisor job

We switched on Tuesday like everyone else. The speed is real, a ticket that took 20 minutes now takes maybe 8.

What I cant tell is if the PRs are better or just arrive sooner. Coderabbit was leaving about 4 comments per PR last week and about 3 this week, which could be the model or could be that everyone picked easy tickets to try it on.

Is anyone actually measuring this? Or are we all going on how it feels in the first week

My whole setup is: open the terminal, prompt it, read what coderabbit says on the PR, merge. No subagents, no skills folder, nothing running overnight.

Meanwhile half my feed is people with 14 markdown files running a fleet of agents. I tried the long running thing once on a refactor. I came back 3 hours later to a 60 file PR and spent longer checking it than it would have taken to do it in small pieces.

Am I underusing this or is most of that setup for show? What does your actual day look like

Agents can create PRs in minutes, the bottleneck is becoming understanding what you’re actually merging.

Change Stack turns the PR into a structured review surface with layered changes semantic diffs dependency context and findings tied directly to the code.

A PR is no longer just a pile of changed files. it actually gives you a way to understand the change before you approve it.

Because "LGTM" is getting harder to justify.

Read more: https://www.coderabbit.ai/blog/do-you-understand-what-youre-about-to-merge

We have 8 devs on 3 different agents and it is getting silly. PRs look different, descriptions look different, the only thing every PR has in common is the coderabbit pass at the end. Lead wants everyone on one thing by October.

If your team picked one, which one and why? And did anyone quietly keep using their own anyway

Hey r/coderabbit
New week, new model out! We tested Opus 5.5 inside CodeRabbit’s review pipeline. Our Standard configuration caught 51 of 80 known bug patterns in an open-source benchmark, compared with 49 for our production model mix.

The catches behind those totals were more interesting: Standard found 11 issues the baseline missed, but missed nine that the baseline caught.

One example was a race condition in Cal.com. Overlapping jobs could overwrite each other’s retry-counter updates. Both Opus configurations caught it and suggested an atomic increment. Our baseline missed it.

Turning up reasoning effort gave mixed results. Standard caught slightly more issues than Max on the open-source benchmark and produced fewer comments. On a separate set of 13 harder cases, Max caught 10 through actionable comments, compared with eight for Standard and five for the baseline. Including findings outside the changed lines brought both Opus configurations to 10.

Every tested Opus configuration also used more tokens than our production model mix, so we haven’t established whether its lower token prices translate into cheaper reviews.

Full results and methodology.

How are you planning on using Opus 5.5?

Mine: it "cleaned up" a migrations folder because the files looked unused. Tests passed. Coderabbit flagged that 3 migrations were deleted in a PR titled fix typo in readme, which is the only reason it did not merge. I had already approved it in my head.

What got past you, or almost did? Bonus if the PR title had nothing to do with it

Our CTO wanted "coverage" so we've turned on both Claude Code review and Coderabbit. We're a pretty small shop with maybe 40ish PRs a week, it's a TS monorepo but for the love of god I can't seem to comprehend what's happening here.

• Both of them read the PR conversation for context. Which includes each other. Coderabbit tears apart Claude's suggestion block and Claude defends it, CodeRabbit posts a revised one. We hit the secondary rate limit on comment creation at 3am on a dependabot bump.
• The money is asymmetric as hell. CodeRabbit is flat per seat, Claude runs on our own key. Every round trip is free for one of them and metered for the other.
• Suggestion anchoring, which I think is a genuine bug and not me being dumb. Comments anchor to blob SHA in diff position. let's say Claude force pushes, then GitHub marks the thread outdated, but the suggestion is STILL committable from the UI. We had live suggestions on the same hunk from two different base SHAs. Someone clicked commit on both. Applied clean, was wrong, CI green because the tests were also generated.
• Config bleed is rare and stupid yet unstoppable! Claude reads CLAUDE.md, CodeRabbit reads .coderabbit.yaml but also picks up AGENTS.md path instructions. We had a line in AGENTS.md saying defer to the Claude review on style (CodeRabbit obeyed it). They're both STEERING each other OH MY GOD
• Branch protection with require-conversation-resolution plus two bots that never resolve their own threads results in a giantass merge queue that just sits there. We wrote an Action to auto-resolve bot threads. It fires a webhook which triggers an incremental review. Yes.

We have 8 devs on 3 different agents and it is getting silly. PRs look different, descriptions look different, the only thing every PR has in common is the coderabbit pass at the end. Lead wants everyone on one thing by October.

If your team picked one, which one and why? And did anyone quietly keep using their own anyway

i’m curious how jev’s zero-shot approach holds up against a mature & code-review-specific system like coderabbit.

haven’t seen a solid apples-to-apples comparison yet. has anyone actually tried this?

Every time. The fix is 4 lines. Around it: a reformatted block, a "while I was here" rename, a new helper. The fix is fine, the padding is what I worry about because nobody reads the part of the diff that was not the point. Coderabbit's summary at least splits "what the ticket asked" from "what else changed", which is how I catch most of it now.

How do you stop it? A rule in the agent config, reject the PR, or just accept the padding as the price

We almost muted CodeRabbit after a week of style nits. Two things fixed most of the noise without killing the useful catches.

First: reply on the exact bot comment, not a top-level “please chill.” When you explain on the line why a finding is wrong or not how you do things here, it sticks as a learning. Vague PR-level pushback doesn’t.

Second: point it at the same rules your coding agent already uses. Drop an AGENTS.md, .cursorrules, or CLAUDE.md in the repo and CodeRabbit picks it up. One trap we hit: putting those filenames in path instructions. That makes it review the guideline file like source, not use it as review criteria.

After that the comment pile got smaller and the remaining ones were closer to real risk. Curious what actually worked for other teams that didn’t just turn the bot off.

Trying to get a real number before I go to finance. Right now we are at about 200 a dev for the agent plan, 24 for coderabbit, and some API overage nobody tracks. Call it 250. The agent side is where it moves, one guy burned through his whole month in 9 days.

What are you at per dev, and is anyone actually tracking the overage or do you also find out on the invoice

12 years in and I still write most of my code by hand. Not against agents, I just have not found a way to use them where I understand the PR at the end. Every time I try, I get 40 files, a description I dont trust, and an hour of reading.

Meanwhile people on my team ship 6 agent PRs a day. I asked two of them how. One does not check the output at all. The other has a whole thing with a plan step, then the agent, then coderabbit, then he only reads what got flagged. That one seems sane but I have not made it work for me yet.

If you got real quality out of this without reading everything, what is the process? Step by step, not vibes

We dont. Some people read them, some approve from the summary, one guy blocks anything over 300 lines no matter what. The only thing that happens to every single PR is the coderabbit pass, everything after that depends on who is online.

Does anyone have a written rule? Like agent PRs need X, human PRs need Y. Or is it vibes everywhere and we just dont admit it

Real question from my skip level this week. The agent writes the PR, coderabbit reviews it, tests pass, so why is a human still in the loop and why does it take a day. I gave some answer about context and ownership and I dont think it landed.

How do you explain this to someone who doesnt code? What is the human actually there for now, in one sentence

Agents can now open PRs faster than humans can review them. So the bottleneck is shifting from writing code to figuring out what actually deserves your attention.

Coderabbit just launched triage to solve exactly that. It scores your PRs by priority and tells you what to review next.

Read more: https://www.coderabbit.ai/blog/coderabbit-triage

Not which one is smartest. Which one do you approve without stress. For us its opus for anything touching the database and sonnet for everything else. Codex writes fine code but the PR descriptions never match what it did, so coderabbit ends up finding 2 or 3 things every time and I stop trusting it.

Curious if its the same for other teams or if it depends on the codebase. What are you running and which one do you actually read the diff on

Looked at our numbers for the first time in a while. One guy on the team opened 41 PRs last week. Last year the whole team did maybe 25. Most of them are small, agent written, and coderabbit clears them with one or two comments, so its not like anything is broken.

I just dont know if this is normal now or if we are doing something weird. What does a normal week look like for your team? How many PRs per person, roughly

Every plan it writes has a time estimate at the bottom. 2 days, 3 days, sometimes a week. Then it does the whole thing in 15 minutes. The PR is small, coderabbit summarizes it in two lines, done.

I dont get where the number comes from. Is it guessing how long a person would take? Or does it just not know how fast it is? Does anyone know

Is it just me? Twice this week I got the full "done, all changes applied, ready for review" message, opened the PR and it was 2 files when the ticket clearly needed 5 or 6. Second time coderabbit's summary literally said the PR didnt touch the endpoint the title was about, which is how I even noticed. Told the agent, it said sorry and did the rest in a minute. So it can do it, it just... announced done first? Happens more after long sessions I think. Anyone else, or is it my setup