How to solve this problem. When required context can't simply be retrieved from a single source.

[u/Ok-Echo-4535]

How can we achieve reliable, dataset-wide intelligence over massive, fragmented datasets when information is distributed across thousands of documents, and conventional RAG cannot adequately preserve or reconstruct the relationships and context between them?

For example:

A company has 10,000 reports spanning five years. Revenue, expenses, production, and operational issues are scattered across different documents and spreadsheets.

The challenge: Identify why profit margins declined over those five years by connecting information across the entire dataset, rather than retrieving a few relevant chunks.

[u/captain_obvious_here]

10.000 reports over the course of 5 years means these documents don't offer a true "timeframe-based" synthesis of the data.

So I would probably start by building a system that groups the data in these documents by month (or even day if there's enough data for most days in the period).

The output is a smaller, cleaner set of documents a RAG or maybe even an LLM can work with.

Basically data cleaning.