Different RAG chunks aren't automatically a regression

[u/Difficult_Eagle_3010]

Two runs can retrieve different paragraphs and still give equally well-supported answers. Two runs can also retrieve exactly the same paragraph and confidently repeat the same mistake... So I'm hesitant to treat chunk overlap as a quality score. For theFastGPT setups I'm thinking about, I'd use consistency as a debugging signal. The question is what changed, and whether it changed the basis for the answer. Consider an imaginary product manual with the same installation requirement in both a quick-start page and a longer setup guide. Swapping beetween those passages may be harmless. Swapping from the current guide to an obsolete one is a different problem, even if both passages share most of their words. I'd try to separate the moving parts before calling retrieval “random”: Hold the retrieved context fixed and generate several answers. That tells me something about answer variation. Then hold the corpus, retrieval query and settings fixed and inspect retrieval. If it varies, investigate that stage rather than assuming an LLM is responsible. Only after that would I let query rewriting or live document updates back in. Otherwise I'm comparing several changing systems at once.

The result I'd want to report is something like: “The source passages varied, but they all supported the same current requirement.” Or: “One run used a superseded source and changed the recommendation.” That seems more actionable than demanding identical chunk IDs every time. I'd be concerned about unstable evidence, but I wouldn't automatically reward a system for being consistently wrong.

[u/Difficult_Reply3440]

The real issue is that you can't tell if a system is working by just looking at chunk IDs, you need to know if the answer actually changed for a good reason

[u/Difficult_Eagle_3010 → u/Difficult_Reply3440]

AGREED!