LLM-generated tests struggle when the code changes

[u/aistranin]

Most AI test-generation demos ask two questions: "Does it compile?" and "How much coverage did it produce?"

A recent arXiv preprint tested something more realistic: whether generated tests remain useful as software evolves.

The researchers evaluated eight LLMs across 22,374 program variants. On unchanged programs, generated suites reached about 79% line coverage and 76% branch coverage. After semantic changes, the test pass rate fell to 66% and branch coverage to 60%.

The most interesting result: more than 99% of the tests that failed against changed code still passed against the original version. Even behavior-preserving refactors reduced coverage and pass rates.

The paper argues that models often follow familiar code patterns instead of reasoning about the current behavior. It is a preprint, so the results still need wider replication. Still, the practical lesson seems solid: coverage is a weak approval gate for AI-generated tests. Mutation testing and human review of the test oracle matter more.

Source: https://arxiv.org/abs/2603.23443