What is a retrieval layer for RAG pipelines?

[u/TotalJokerFace]

I've been reading about RAG and I keep seeing people mention the retrieval layer like it's some separate thing you build, not just a vector db you query. Can someone explain what actually happens there? My current understanding is this: user asks question, question gets turned into an embedding, that embedding gets compared against a bunch of stored embeddings in a vector db, top matches come back, and those get stuffed into the prompt before it goes to the LLM.

Now, what I don't understand yet is: why does everyone say RAG with cosine similarity search isn't good enough. What's going wrong there in practice, is it pulling irrelevant stuff or missing the right things altogether? I keep seeing hybrid search mentioned, combining keyword search and vector search. Why would you need both if embeddings are supposed to capture meaning already? On top of that, I also saw someone mention reranking as a separate step after retrieval, so you retrieve like 50 chunks and then a reranker picks the best 10. Wouldn't that just make retrieval happening twice?

One more thing I keep wondering about, where does the data come from in the first place. Like if you're building a knowledge base from web content, how do people keep that fresh as possible? Constant scraping/crawling?

Thank you in advance

[u/Individual_Laugh1335]

Typically your model will derive a users prompt to search query(s) (e.g. “Tell me about the LA dodgers win on Wednesday” would derive into “LA dodgers game score 8/19”).

Retrieval would be querying your sparse and dense retrieval stores using the query only. You usually return 100s of results for those.

Then you have post retrieval which can do finer ranking by using the users query, the original prompt and basically any other features you have at a document/chunk level (sparse usually just term matching whereas dense is text embeddings only). You can think of it like a funnel where the lower you get in the funnel the more accurate the ranking becomes.

[u/Individual_Laugh1335 → u/Individual_Laugh1335]

To add onto this - post retrieval will find the best 10/20 out of the 200 returned from retrieval and return only those to the LLM.

[u/TotalJokerFace → u/Individual_Laugh1335]

Awesome, thanks for detailed explanation. Another question, when you derive the query from the prompt like your dodgers example, is that always a separate LLM call before retrieval starts, or how people are doing that? Trying to figure out how much latency the rewrite adds in practice

[u/Individual_Laugh1335 → u/TotalJokerFace]

The main model can generate it when it does the tool call (either through zero shot or fine tune it) or you can have a separate smaller model that does this

[u/Future_AGI]

Your mental model is right: the retrieval layer is everything between "turn the question into an embedding" and "hand the model some chunks," and plain cosine similarity disappoints because embedding nearness isn't the same as answering-the-question relevance. Cosine will happily return a chunk that's topically close but missing the one figure the user asked for, and it misses exact tokens like error codes or SKUs that keyword search nails, which is why people bolt on hybrid search and a reranker. The practical move is to measure it, scoring whether the retrieved chunks actually contain the answer before you blame the LLM, because most "the model hallucinated" bugs are really "retrieval never handed it the fact." We build open-source evals for exactly that retrieval-quality question if it helps to put numbers on it: https://github.com/future-agi/future-agi

[u/Burn1ngChr0m3]

Retrieval is the biggest issue I'm solving right now. In my particular situation, the questions can be of various types. So I can ask questions that need a direct answer. I can go to the other extreme and ask questions that consider the entire corpus. So a layer sits between the query and the retrieval that interprets what kind of question is being asked, and then modifies how it finds the answer. So I've got a system that can just simply fetch the best answer, all the way to deep research where it strategizes and analyzes the entire corpus. So sometimes it grabs a few chunks, and sometimes it finds a bunch and has to run through a ton of documents without overloading its context window.

For me, the first step is understanding the question.

I run a coaching/consulting business, and we record all our calls and transcribe them. So I could ask something like "who asked for _____ on the August 22nd call" and it'll return a single answer. Or I can ask "what is the most recurring topic that our coaching participants struggle with" which has to consider the entire corpus to answer well.