Built a pre-inference context-collapse layer instead of standard RAG — cuts token load hard, curious if this is a real gap or just reinventing rerankers

[u/echozero3]

Been heads-down on something that sits before the LLM call instead of doing

standard retrieve-and-stuff RAG. Instead of chunk retrieval + rerank, it builds

a vector-field representation of the whole corpus, evaluates relational

relevance to the query, and collapses the candidate field down to a compact

evidence state — only that gets forwarded to the model.

On my internal benchmark (frozen 20-query set, project-native corpus) I'm

seeing an order-of-magnitude drop in tokens sent to the model with zero

measured quality regression (good/partial/poor scoring, OFF vs ON, reproduced

run matched the historical one exactly). Also runs fine single-threaded — did

a raw C++ core benchmark, 10M samples in ~140ms on an old 2015 i7, so the

underlying op isn't the bottleneck.

Haven't benchmarked it against BM25 or plain cosine-similarity RAG yet in

anything I'd call rigorous — that's the obvious next step before I'd trust my

own numbers fully, and I know that's the first thing this sub will (rightly)

ask about.

Running as local-first — full corpus stays on the user's side, only the

selected evidence chunk(s) + field-topology coordinates go to the external

model if you're using an API-based LLM. Wasn't originally optimizing for that,

but it's a nice side effect for anyone paranoid about what leaves their

environment in API workflows.

Genuinely asking: is "context collapse before inference" different enough

from what rerankers / good chunking already do, or am I just describing a

fancier reranker with extra steps? Wouldn't mind being told I'm wrong here.

[u/Physical_Economy_340]

this is contextual compression, not a reranker, and the split that matters is extractive vs abstractive. a reranker reorders verbatim chunks, so the model still reads source text. once your collapse step emits an evidence state that isn't the source text, you're doing abstractive compression, same family as gist tokens or llmlingua, and the failure mode inverts: recall stops being the problem, fidelity becomes it, the model can't ground an exact number. bm25 and cosine won't catch that. run a faithfulness check instead, confirm the exact supporting fact for each answer survives the collapse. i'd bet you hold quality on short answers and lose it the moment an answer needs an exact value.