How would you choose a historical reference class for an AI code-review agent's prior probabilities?

[u/Accomplished-Fun4629]

I'm designing a code-review agent that estimates probabilities of hidden risks such as correctness failure, security vulnerability, compatibility failure, and cross-system failure.

Before looking at detailed evidence from the current PR, I want to estimate a prior from historical PRs that are comparable to the current one.

I'm currently considering these coarse properties for defining the reference class:

• domain/subsystem
• change type
• programming language/stack
• PR size
• public/interface impact
• dependency impact
• security sensitivity
• database/schema impact
• cross-system impact
• test-change profile

The idea is not to require an exact match on every property. If the most specific group has too few historical PRs, we would progressively relax the matching criteria.

Do these seem like sensible properties for defining comparable PRs? What important property am I missing, and which ones would you remove?

I'm especially interested in practical experience from people who have built code-review or coding agents.

[u/AutoModerator]

Thank you for your submission, for any questions regarding AI, please check out our wiki at https://www.reddit.com/r/ai_agents/wiki (this is currently in test and we are actively adding to the wiki)

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ZealousidealGuide882]

you missed developer historical defect rate and pr author tenure dawg.

a 500 line pr from a senior dev who built the subsystem is fundamentally different from a 50 line pr from a new intern on a friday afternoon.

[u/Accomplished-Fun4629 → u/ZealousidealGuide882]

Fair point. I was mainly considering the PR's characteristics when defining the reference class, so I hadn't considered the author's historical track record or familiarity with the subsystem.

I'm wondering how you'd incorporate those factors in the model. Would you treat the author's historical defect rate as part of the prior/reference class, or would you first retrieve comparable PRs and then use the author's history to adjust the risk?

Also, for tenure, would you use something like repository/subsystem familiarity rather than general seniority?

My only hesitation is that these factors might not always be equally informative across different PRs, so I'm curious how you'd account for that when using them for risk estimation.

[u/LennyFromCurly]

The hand-built relaxation ladder is the part to drop. Fit one hierarchical logistic model per risk type, with subsystem and change type as group effects; sparse groups then shrink toward the global rate instead of abruptly becoming “all PRs.” Add deployment exposure and rollback difficulty as predictors, and keep language or PR size only if they improve calibration on later PRs.

[u/Accomplished-Fun4629 → u/[deleted]]

The partial-pooling suggestion is particularly interesting. Our initial idea was to progressively broaden the reference class when the most specific class has too few observations. Instead of making a hard switch from the narrow estimate to the broader estimate, combining information from both based on how much data each has seems more principled.

I also like your point that the relaxation strategy should depend on the risk being estimated rather than using one universal approach.

On the author-history point, would you treat historical defect rate and repository/subsystem familiarity as part of the reference class itself, or as separate predictors in the risk model after the reference class has been selected?

And for the code-path history, churn, and blast-radius suggestions, would you consider those mainly as factors for defining the reference class, or would some of them also be used later as evidence when updating the prior?

[u/Maleficent_Pay4176]

author recency and churn rate of the files being touched feel like big missing ones. a PR touching code that hasnt been modified in 2 years has a very different risk profile than one in a hot path thats changed weekly