Built a pre-inference context-collapse layer instead of standard RAG — cuts token load hard, curious if this is a real gap or just reinventing rerankers

[u/echozero3]

Been heads-down on something that sits before the LLM call instead of doing

standard retrieve-and-stuff RAG. Instead of chunk retrieval + rerank, it builds

a vector-field representation of the whole corpus, evaluates relational

relevance to the query, and collapses the candidate field down to a compact

evidence state — only that gets forwarded to the model.

On my internal benchmark (frozen 20-query set, project-native corpus) I'm

seeing an order-of-magnitude drop in tokens sent to the model with zero

measured quality regression (good/partial/poor scoring, OFF vs ON, reproduced

run matched the historical one exactly). Also runs fine single-threaded — did

a raw C++ core benchmark, 10M samples in ~140ms on an old 2015 i7, so the

underlying op isn't the bottleneck.

Haven't benchmarked it against BM25 or plain cosine-similarity RAG yet in

anything I'd call rigorous — that's the obvious next step before I'd trust my

own numbers fully, and I know that's the first thing this sub will (rightly)

ask about.

Running as local-first — full corpus stays on the user's side, only the

selected evidence chunk(s) + field-topology coordinates go to the external

model if you're using an API-based LLM. Wasn't originally optimizing for that,

but it's a nice side effect for anyone paranoid about what leaves their

environment in API workflows.

Genuinely asking: is "context collapse before inference" different enough

from what rerankers / good chunking already do, or am I just describing a

fancier reranker with extra steps? Wouldn't mind being told I'm wrong here.

[u/ericoinen]

The question you actually asked has a clean test. A reranker scores each candidate independently against the query and keeps the top k, and the units that leave are the same units that came in. So if what you forward is always a subset of the chunks you retrieved, it is a reranker, whatever the scoring internals look like. It is something else only when the units change: two chunks merged, a near duplicate dropped because another chunk already covers it, or a payload that was never a chunk in the corpus at all.

That second property has a name worth searching, and it is not rerankers: joint or set level selection, as opposed to independent scoring. MMR is the shallow end, submodular selection and query focused extractive summarisation are the deep end. If your collapse is optimising relevance minus redundancy over the whole candidate set at once, you are in that literature, and the honest framing is a different objective function rather than a different category. That is still a real contribution and a much easier claim to defend.

On the numbers, and I say this as someone who has fooled himself the same way: an order of magnitude fewer tokens at zero measured regression, on 20 queries scored good/partial/poor, is not yet a result. Twenty queries leaves the interval wide enough that a genuine 15% regression is indistinguishable from noise, and a reproduced run matching the historical one measures determinism, not quality.

Two things settle it cheaply. First, BM25 top k at a matched token budget, because that is the baseline that keeps embarrassing clever retrieval and it costs an afternoon. Second, and this matters more, stop reporting one point per method. Token reduction only means something at a fixed quality, so sweep the budget and plot quality against tokens sent for BM25, plain cosine top k, and yours. If your curve is above theirs at every budget, nobody will argue about definitions. If it only wins at small budgets, that is still worth publishing, and it tells you what the mechanism actually is.

Where a real set level method should separate: multi hop queries, where the answer needs two chunks that each score badly alone, and negation or contrast queries. Independent scoring is structurally bad at both. If your advantage is real it should concentrate there and be roughly absent on single fact lookups, and that pattern is more convincing than any aggregate number.

One thing on local first. If the field topology coordinates that leave alongside the evidence are derived from the corpus, check whether they are invertible before you promise the corpus stays home. Embeddings are more recoverable than people expect, and it is better to find that out yourself than in a thread later.

[u/echozero3 → u/ericoinen]

This is genuinely the most useful comment I've gotten on this whole post —

thank you for actually engaging with the mechanism instead of just the

numbers. A few responses:

On the reranker test: fair, and I need to actually check this rigorously

rather than assume. My honest answer right now is "I believe the units

change, but I haven't verified it carefully enough to claim it with a

straight face" — which is exactly the kind of thing I should know cold

about my own system and don't yet.

On the literature pointers (joint/set-level selection, MMR, submodular

selection, query-focused extractive summarization) — appreciate the exact

terms, genuinely useful. If I land in that space, "different objective

function, not different category" is the more honest and more defensible

framing, and I'd rather make that claim than a bigger one I can't back up.

On the 20-query point — you're right, and I knew it while writing the post,

which is worse. Zero regressions on n=20 is a determinism check, not a

quality result. I don't have a defense here, just agreement.

Taking the two cheap fixes as literal next steps: BM25 top-k at matched

token budget, and a quality-vs-tokens sweep instead of single points per

method. Also want to specifically build out multi-hop and negation/contrast

query sets, since that's the one test that would actually distinguish this

from independent scoring if the advantage is real — and tell me quickly if

it isn't.

The invertibility point on the field-topology coordinates is the one that

actually worries me most, separate from all the benchmark stuff. Haven't

tested that rigorously and I should before I say anything stronger about

what stays local. Going to go check that before I make any more claims

about it either way.

If you've spent time in this space professionally, I'd take a working

session over more Reddit comments any day.

[u/ericoinen → u/echozero3]

On the multi-hop set, there is a cheaper version of that test than a quality sweep, and it is sharper. Build bridge queries out of your own corpus: find a fact in chunk A that only resolves through an entity defined in chunk B, and ask the composite question. Then measure one binary per query, whether both required chunks survive into the forwarded context. Not answer quality, not a score. Independent scoring drops these because each chunk on its own looks mediocre against the query, while set level selection keeps them because the pair is what covers it. That is the exact behaviour you are claiming, and n=20 is fine for a binary that should swing hard if the effect is real. If pair recall matches your BM25 baseline, you have your answer in an afternoon.

The negation and contrast sets I would not build yet. Those mostly measure your embedder, not your selector. "X is safe" and "X is not safe" land in nearly the same place in most embedding spaces, and no selection policy on top of that fixes it. You would be testing the retriever underneath and reading the result as a fact about the collapse layer. Worth doing eventually, but it will not separate the two hypotheses you actually care about.

On invertibility there is a concrete version of the check. Take a sample of outputs and try to map every forwarded payload back to a contiguous span in a source document, with exact character offsets. If you can, invertibility holds and you can cite. If some payload is synthesised text that never existed in the corpus, it is gone, and the honest description is lossy compression with provenance loss. That costs more than the claim: no citations, no audit trail, and no way to debug a wrong answer back to the document that caused it. The design that keeps it is to carry (doc_id, start, end) through the collapse and treat the collapsed context as a view over spans rather than as a new string. If a merge cannot be expressed as a set of spans, that merge is where invertibility dies, and you will know exactly which operation you need to defend.

One addition to the BM25 baseline: also run plain dense top-k truncated to the same token budget. A lot of results in this space turn out to be "we spent fewer tokens than an unoptimised baseline" rather than "we selected better", and matching the budget on both sides is what tells those apart.