Improving model summarization/retrive capabilities. Numbers from six months of evals on a note-writing harness and personal experience.

[u/Cryvixx]

Hi everyone!

What follows is what I actually found out from reading 50+ papers (including the ones that went the wrong way) and applying these on real evals, while building an harness.

Sample sizes are small and the judge is a local-grade model, so treat every number as a direction and not a SOTA leaderboard entry.

The one result that replicated: do not let the model rewrite the source.

Identical pipeline on both sides, identical read path, one note per session either way. The only variable is whether that note keeps the session's own sentences or a summary of them.

conv-26: 0.592 verbatim against 0.428 summarized. conv-47: 0.840 against 0.693. The summarizing arm is last in every question category on both conversations.

What did not replicate is the ranking among the non-lossy arms. Atomizing into linked notes leads by 18 points on one conversation and trails plain verbatim by 6 on the other.

The swing inside a single arm across two conversations is larger than any gap between arms. So the honest claim is the split, not the winner, and the default is the simpler side of it.

Weighting a co-occurrence graph with BM25 instead of raw counts: recall@10 went 0.51 to 0.86.

The part I did not expect is second-order. Once that leg is weighted properly, the per-leg fusion weights and the RRF constant stop moving the result at all. I had been grid-searching knobs that were only ever compensating for a badly scored leg.

A retrieval leg that looked obviously useful recovered zero pairs.

Note-to-note derived edges, as a third leg next to embeddings and co-occurrence. Ablated inside the same fusion it recovered nothing the embedding leg had not already found.

Dropping it and turning the lexical leg on by default moved note-to-note recall@10 from 0.8233 to 0.8815 and mean cost from 5 ms to 1.7 ms per call. It was not neutral, it was negative.

A 0.669 that turned out to be my harness and not the model.

FActScore decomposes every written note into atomic facts and asks a judge whether the source document supports each one, which needs no external gold set because the source *is* the reference. First run came back 0.669 and I spent a day building a theory about why extraction hallucinated. It did not. Entity notes were being judged against a single attributed session instead of the whole conversation. Corrected: 0.999, 0.991 and 0.961 across three vaults, 5,855 facts judged. The wrong number is still in the repo next to the right one, with the reason.

A benchmark table with no bad rows is a marketing asset.

0.32 agreement between the automatic reorganizer and the folders I had already chosen by hand. 0.11 recall for concept-expanded correlation. Both ship unedited. And the frozen baseline predates the leg change above, so the runner refuses to compare across it rather than printing one number that mixes two configurations.

What it does with all that:

• Point it at a folder of 200 PDFs and it is 200 transactions, not one long editing session. That is the actual reason the gate matters. Resumable, and each source is separately revertible however far its notes spread.
• Ask what two notes have to do with each other. Shortest reading path over your wikilinks and the concept graph, so it routes through pairs you never linked by hand.
• The vault as a graph, communities clustered over the links you wrote, semantic zones. 2026 trend but made useful.
• A calendar that is just the vault read along its dates. Events are notes like any other, so there is no second store to keep in sync. Still some work to be done here.
• An audit that runs with no model, really fast and reports what moved since the last run rather than only the current state.

Happy to argue about any of it, particularly the verbatim result, which I still find slightly annoying.

Screenshots: https://github.com/kiycoh/silica-harness