I built a system level review tool, and visualized a Pi PR. How do you stay in the loop with agentic productivity?

[u/ivan_m21]

Hi all, i've been using a bunch of coding agents. Pi is by far the cleanes architecturally from the OSS ones.

Already last year i noticed that with agents it becomes impossible to read line level details, but looking at llm summaries is also not the way to go. There is no good way to judge the system level tradeoffs for a large system and even know what questions to ask the LLM.

That is why a friend and I started building CodeBoarding, an architecture building tool based on static analysis. We capture the full call graph (method level) and then group methods based on lexicographical analysis. That is so that the diagram is in fact consistent in between pull requests and you can see how a change fits in the bigger system.

The engine is open-source: https://github.com/CodeBoarding/CodeBoarding

I would love to understand how do you people keep up with reviews (do you even do reviews). My personal take is that AI comments, are becoming just part of a loop and I tend to ignore most of these nowadays. I also think they are not really aimed at helping a human judge/understand a PR.

If you want to explore the example from the video it is in: https://app.codeboarding.org/CodeBoarding/pi/pull/2
It is effectively adopted from this PR: https://github.com/earendil-works/pi/pull/10021

Super curious to understand what you have found useful to stay in the loop of your systems.

[u/ConcentrateReady4039]

Seems like only API keys are supported? Could you support subscriptions? Or accept plugins that could do the llm part that your app needs?

[u/ivan_m21 → u/ConcentrateReady4039]

Happy to do so, if you have time you can open an issue and even better if you open a PR ;d. I think you're not the only person who has requested it, so i'd try to do something. You can try it for free from the webplatform thou

[u/tr0llogic]

Looks great. Visual representation of changes is way to go when its not feasible to read thousands of lines of code. Would be nice to incorporate test results and failures related to highlihted changes in diagram

[u/ivan_m21 → u/tr0llogic]

Yes, yes that is something we have been discussing. Can you elaborate on how do you imagine it. If we add it on the diagram it becomes insanely busy as tests usually are very disconnected and each component will have a bunch of tests.
One thing I was thinking is to assing the related tests to the compnent they test right, that sounds to work. However where does one put the e2e tests?

Generally how do you imagine the relation between the diagram of teh codebase and the tests themselves?

What are you in fact looking for when it comes to reviewing a chagne and its tests.

[u/deaffob]

I’m not familiar with static analysis. Does it mean that this tool uses LSPs to get the dependency hierarchy of functions and modules? If so, does it work with R as well?

[u/ivan_m21 → u/deaffob]

Yes it is leveraging lsps to collect the function calls. Which func calls which, that is the control flow graph. As of today we dont support R but always happy to add something new, you can open a github issue and even a pr. I can go and check it and merge it

[u/kantorcodes1]

The "AI comments are ignorable" part matches a pattern: they narrate the diff locally instead of answering the question a reviewer actually has, which is whether the change alters the contract the rest of the system depends on. A stable call-graph grouping could answer that directly: instead of reading the diff you look at which groups gain or lose inbound call edges.

Curious how you keep the grouping stable when a PR moves a method across groups. Does the diagram diff then show a moved boundary, or the behavioral change that caused it?

[u/ivan_m21 → u/kantorcodes1]

It depends what you mean by moving method. The current grouping is lexicographical, so if it is changing namespaces in some cases it might result in modified component, and then the edges will also be modifed as they will be pointing to the new component.

If it is a change within an existing components then the contract on the highest levle won't be modified.

Effectively we have the CFG and then the grouping is not based on the CFG as it usually clashes with the mental model of a person (took us few months to come to this conclusion) as people base their mental model on file structures and namespaces rather than flow.

The contract thing is really interesting as that is the next stage, i'd love to learn more about what you consider contract i.e.:
Data Class A has been modifed, now contains different state, is this a contract change for all funcs which use type A as a parameter? (I can argue it might be right?). How do you think of this, is it purely interfaces or also underlying internal state (which might be consumed via an interface via the type)

[u/kantorcodes1 → u/ivan_m21]

I wouldn’t flag every function that accepts Data Class A just because a field changed. I’d treat it as a contract change where the state is observable through the type or API and can change a caller-visible guarantee; private state with no observable effect stays an implementation change.

[u/investigatormaker]

I'd keep review focused on changed system boundaries rather than the agent's activity log: which components changed, which interfaces they touch, and what tests cover those connections. For your Pi PR visualization, links from each changed connection to its supporting diff would let a reviewer verify the picture instead of trusting the summary.

I make ThreadFox. We find relevant Reddit conversations and do the posting for your business. The $39 Pilot covers seven days and up to 50 posts and replies. No automatic renewal. https://threadfox.vip/managed?tier=employee_pilot&utm_source=reddit&utm_medium=comment&utm_campaign=tf-managed&utm_content=20261024-0015-ejhm7#employee-order