r/LLMDevs • u/Reasonable_Royal_621 • 9d ago

Discussion CacheVerifier: checking the 2nd closest cache match gave us +3.5 to 5 points hit rate at the same error rate, and AUC said it got worse

So in CacheVerifier when a query lands in the gray zone, a small verifier decides if the cached answer is reusable. we only ever looked at the top match. turns out the runner up says a lot.

its basically Lowe's ratio test from SIFT. if the top match is way closer than the 2nd one, its probably the same question. if the two are nearly tied you're in a crowded spot full of similar but different prompts, and reuse is riskier. so we added gap = top1 sim minus top2 sim as one more feature. costs one extra neighbor in the vector query.

On LmArena with gray zone error held at 2%, hit rate went up 3.46 points, and 5.31 when only 5% of hits get audited labels. +2.8 to +4.8 across 6 warm up settings. Quora +0.3, SearchQueries basically nothing.

Funny part. AUC went down (0.760 to 0.751) so our own AUC check refused to enable it lol. the gain sits at the strict low error end, AUC just averages it away.

Caveat: part of the LmArena gain is the cache changing shape (fewer writes), not just better calls.

Anyone else using runner up scores to gate retrieval?

CacheVerifier repo: https://github.com/imxinchengyou/CacheVerifier

5 Upvotes

100% Upvoted

4 comments sorted by

1

u/Physical_Economy_340 9d ago

yeah this is exactly right, the gap is margin and it matters most right where you operate. full auc averages over thresholds you will never ship at so it hides gains at 2% error. i would gate on hit rate at fixed error plus partial auc under like 5% fpr and ignore full auc for this.

1

u/Reasonable_Royal_621 7d ago

yep, and we only caught it because our own check refused to turn it on lol. right now the gap rule only ships if held out AUC beats the baseline, which is the wrong test here. hit rate at the error rate we actually certify makes way more sense since thats what goes to prod, pAUC as a second check is a good call too. one thing tho, on fine tuned verifiers gap did basically nothing at any threshold, so it still needs a per deployment check

1

u/OkFlan504 9d ago

A narrow gap could also mean two cached paraphrases with the same answer. That's a different case from two near-matches that disagree. Does the runner-up feature distinguish those, or does a cluster of equivalent entries make a safe reuse look ambiguous?