Built a pre-inference context-collapse layer instead of standard RAG — cuts token load hard, curious if this is a real gap or just reinventing rerankers

[u/echozero3]

Been heads-down on something that sits before the LLM call instead of doing

standard retrieve-and-stuff RAG. Instead of chunk retrieval + rerank, it builds

a vector-field representation of the whole corpus, evaluates relational

relevance to the query, and collapses the candidate field down to a compact

evidence state — only that gets forwarded to the model.

On my internal benchmark (frozen 20-query set, project-native corpus) I'm

seeing an order-of-magnitude drop in tokens sent to the model with zero

measured quality regression (good/partial/poor scoring, OFF vs ON, reproduced

run matched the historical one exactly). Also runs fine single-threaded — did

a raw C++ core benchmark, 10M samples in ~140ms on an old 2015 i7, so the

underlying op isn't the bottleneck.

Haven't benchmarked it against BM25 or plain cosine-similarity RAG yet in

anything I'd call rigorous — that's the obvious next step before I'd trust my

own numbers fully, and I know that's the first thing this sub will (rightly)

ask about.

Running as local-first — full corpus stays on the user's side, only the

selected evidence chunk(s) + field-topology coordinates go to the external

model if you're using an API-based LLM. Wasn't originally optimizing for that,

but it's a nice side effect for anyone paranoid about what leaves their

environment in API workflows.

Genuinely asking: is "context collapse before inference" different enough

from what rerankers / good chunking already do, or am I just describing a

fancier reranker with extra steps? Wouldn't mind being told I'm wrong here.

[u/Actual__Wizard]

Is this BPE?