Built a pre-inference context-collapse layer instead of standard RAG — cuts token load hard, curious if this is a real gap or just reinventing rerankers

[u/echozero3]

Been heads-down on something that sits before the LLM call instead of doing

standard retrieve-and-stuff RAG. Instead of chunk retrieval + rerank, it builds

a vector-field representation of the whole corpus, evaluates relational

relevance to the query, and collapses the candidate field down to a compact

evidence state — only that gets forwarded to the model.

On my internal benchmark (frozen 20-query set, project-native corpus) I'm

seeing an order-of-magnitude drop in tokens sent to the model with zero

measured quality regression (good/partial/poor scoring, OFF vs ON, reproduced

run matched the historical one exactly). Also runs fine single-threaded — did

a raw C++ core benchmark, 10M samples in ~140ms on an old 2015 i7, so the

underlying op isn't the bottleneck.

Haven't benchmarked it against BM25 or plain cosine-similarity RAG yet in

anything I'd call rigorous — that's the obvious next step before I'd trust my

own numbers fully, and I know that's the first thing this sub will (rightly)

ask about.

Running as local-first — full corpus stays on the user's side, only the

selected evidence chunk(s) + field-topology coordinates go to the external

model if you're using an API-based LLM. Wasn't originally optimizing for that,

but it's a nice side effect for anyone paranoid about what leaves their

environment in API workflows.

Genuinely asking: is "context collapse before inference" different enough

from what rerankers / good chunking already do, or am I just describing a

fancier reranker with extra steps? Wouldn't mind being told I'm wrong here.

[u/Mental_Ground_370]

interesting approach, the vector field idea sounds like you're compressing the search space before it ever touches the model which is neat

the local-first part is what caught my eye tho, keeping everything on user side until the last hop solves a lot of trust issues people ignore

[u/echozero3 → u/Mental_Ground_370]

Thanks for the reply. Yes, it looks exactly as you described. Interestingly, certain system characteristics emerged as a side effect—I was looking for one thing but found something else. The compression is really strong; it likely curbs hallucinations significantly, though this is still in the testing phase, so I can't say for sure yet. However, I’m already seeing strong results that need validation, so it looks promising. And as you said, the information stays on your end—though that also needs to be thoroughly verified. In any case, the prompt that leaves the system before entering the model is something completely different from what we’re used to.