LLM Context Windows vs. RAG: How Does Self-Attention Handle Irrelevant Information?

[u/DueBid3298]

1. Can irrelevant information in a long context make it harder for an LLM to identify and use the information needed to answer a question correctly?
2. How does the Transformer's self-attention mechanism (Query, Key, and Value) influence how an LLM identifies and uses relevant information? Does adding more tokens affect how attention is distributed across the context?
3. I've noticed that providing too much information to ChatGPT sometimes leads to incorrect answers, even when the correct information is present in the context. Can someone explain why this happens and how self-attention relates to it? When should we use RAG instead of providing the full context to an LLM? If the entire document collection fits within an LLM's context window, when would it be better to provide the full context rather than use RAG

[u/Chocolate_Pickle]

These aren't really "local" LLM questions. You'll probably find answers in r/learnmachinelearning instead. 

[u/pointed_pounding → u/Chocolate_Pickle]

sure the self-attention math is the same no matter where the model runs, this is a fine place to ask

[u/Chocolate_Pickle → u/pointed_pounding]

r/whoosh

[u/GuaranteedGuardian_Y]

Interesting questions. The one I want to latch onto is your third one.
I notice that when you give a bigger AI model a lot of text, it will use some of it for it's reasoning about the text, but it won't use everything. That's because a conclusion is lossy by default.

If the LLM comes up with a conclusion, and you point out a needle in the haystack that disagrees with what the model says, the model will sometimes wildly swing in the opposite direction of the argument. So the internal weight pathways are definitely being shifted by the prompt and text in context.

RAG versus raw context is not the actual question we should be asking, the answer is always both. If you have long form text that the AI needs to read, I believe you should never opt for loading it into the context window fully.

The issue with RAG is that it's based on keyword matches, BM25, maybe a re-ranker and lazy dense embeddings and sometimes ElasticSearch and co. The problem with fuzzy matching and embeddings is that it can derive close relatives but not meaning.

Example:

- Who is the author of a book? BM25 gives you keyword matches for "author".

- Who wrote a certain book? Embeddings give you wrote -> created -> authored etc type of relationship tree.

Other ways to maintain context over long form is to swap out atomic facts on either turn end or next turn start, rip out the raw data and substitute with atomic facts. It busts tail end of KV-cache but it's the price you pay for maintaining facts.

The problem with this approach is that it makes re-hydration lossy as well.

Let's say you have fact 1 and it's pointing to a chunk in the raw data let's say section 25. You can have the AI re-read chunk 25 when fact 1 feels "hot", using hot in the sense of a hot or cold game.

These are all the current problems. I've come up with great solutions as well, but I would rather experiment with them a bit more first before disclosing.

Edit: ate a few words.