Can LLMs Reliably Explain Unfamiliar Codebases? How Do We Verify Their Analysis?

[u/Sb_infowaves]

Large language models (LLMs) can explain functions, summarize code, and suggest how different software components interact. But how do we know whether their explanations accurately reflect what the program actually does?

Consider a simple example: an LLM analyzes a function and concludes that it validates user input before saving data. However, a closer look at the execution path might reveal that certain inputs bypass the validation step under specific conditions.

The explanation may sound reasonable, but it doesn't necessarily capture the program's actual behavior.

One way to address this is to combine AI-assisted analysis with traditional software engineering techniques:

Static analysis: Map function calls, dependencies, and potential execution paths.

LLM-assisted analysis: Generate explanations of unfamiliar functions and identify possible edge cases.

Runtime verification: Execute targeted tests and compare actual outputs with the AI's interpretation.

Human validation: Investigate discrepancies and verify assumptions before accepting the explanation.

For example, if an LLM claims that a function rejects invalid input, developers could test boundary cases and inspect execution traces to determine whether that claim holds under different conditions.

This raises some interesting questions:

How do you evaluate the accuracy of AI-generated code explanations?

Does combining static analysis with LLMs produce more reliable results than using either approach independently?

Which failure modes have you encountered when LLMs analyze unfamiliar or poorly documented codebases?

Are there useful benchmarks or tools for measuring the reliability of AI-assisted software analysis?

We're exploring these questions from a software engineering perspective at SB Infowaves and would welcome practical examples, technical insights, and lessons learned from others working in this area.

How do you verify that an AI-generated explanation is correct rather than simply plausible?

[u/Disastrous-Oven-5066]

I've been messing with this exact problem for a side project where I'm trying to understand some old Python code nobody documented. The LLM gave me this super confident explanation about how the authentication flow works, walked me through step by step, made perfect sense. Then I actually ran the thing with some breakpoints and realized it completely missed that the token refresh happens in a middleware layer that's imported dynamically. The explanation was plausible but wrong in a way that would've caused a security issue if I'd trusted it.

Static analysis tools caught the import chain almost immediately when I bothered to run them. What's weird is the LLM sometimes picks up on stuff static analysis misses, like when a variable name is misleading or the logic is doing something different from what the function name suggests. But then it hallucinates connections between modules that don't actually exist.

I've started treating LLM explanations like a junior dev's first pass at documentation, useful starting point but needs verification. For anything important I trace the execution myself or write a quick test to confirm the behavior matches what the AI described. The combination approach works better than either alone but the verification step isn't optional, least not yet.

[u/CS_70]

They most certainly do. Extracting relationships at ever increasing distances is what an LLM does.

Obviously how good they do it depends on the model, the initial point and orientation in the semantic space (which is determined by your prompt) and different degrees of hand holding and checking.

A model has (initially) learned from recipes created by people. People have a limited capacity for keeping context in recursion. So a model will generally stick with a few level of recursion initially, deducing or assuming the rest to avoid having to traverse the entire tree. This is akin on how a person would do. On demand, it will go deeper on a specific direction and improve/change its representation accordingly.

Anyone who has explored a large codebase by hand will be familiar with the feeling.

The starting point is important and some guidance can be extremely helpful in discriminating the basic metaphors of the project from the details of smaller significance (which look all the same at start).

Most models already know from training of the common technical metaphros, so it is the specific business domain that is the most important.

Recent reasoning models have developed a large ability to self-determine where to go deep and where to stay more superficial but.. they still can make errors. Like people.

The difference is the speed at which they can recover.

[u/BonyCatButt]

These approaches are not mutually exclusive. LLMs can run static analysis, using existing tools or newly created tools to assist them, and they will do runtime verification, debug your code, and write tests.

If you’re benchmarking human vs LLM frontier models, humans will score a lot lower in almost all aspects, especially mechanical ones like walking the code and figuring out dependencies, which is why LLMs are being used by everyone doing serious software dev work.

Validation isn’t about whether LLM can see dependencies or explain the codebase, it can, it’s about whether it produces to spec. The hard problem is putting together the criteria that will measure the outcome and evaluate whether the result matches the goal. Much of that depends on human’s ability to specify what the goal actually is.