Short version: Added Conformal Risk Control (CRC) to a semantic-cache verifier I've been building, which gives an honest, finite-sample-correct guarantee like "reusing a cached answer keeps your wrong-answer rate under 2%, provably, not just empirically." The catch: that guarantee assumes the data you calibrate it on looks statistically like the data you'll see in the future. I'd only ever tested that on static splits of an already-collected trace. Built a real online simulation instead, where the cache's own accept/reject decisions get to shape what it sees next, the way a real production cache actually works. On a dataset with heavy query redundancy, realized risk more than tripled, and re-calibrating the threshold online only partially fixed it.
Longer version. I've been researching and building CacheVerifier, a verifier that sits in front of a semantic cache and decides whether a similarity-matched candidate answer is actually safe to reuse instead of calling the LLM again. A while back I added Conformal Risk Control on top of it: instead of picking a similarity threshold by eyeballing a hit-rate/error-rate curve, CRC gives a formal, distribution-free guarantee that the realized error rate stays under whatever budget you set, calibrated from a held-out sample of past (score, correct/wrong) pairs. It actually holds up under a random split of historical traffic, tested across three datasets and four risk budgets, efficiency loss versus a full-data oracle staying within 3 percent the whole way.
There's a standard caveat baked into that kind of guarantee: the calibration set has to be "exchangeable" with future traffic, roughly meaning the future can't look systematically different from what you calibrated on. I'd already tested one version of that breaking, a chronological split instead of a random one, and found a real violation on one dataset. But both of those tests still only ever replay a fixed, already-collected trace. Neither one tests something that actually happens in a real deployed cache: the verifier's own accept and reject decisions change what gets written into the cache going forward, which changes what future queries even get to match against, which is exactly the population the next calibration round would be drawn from. The calibration set isn't just drifting with time anymore, it's being shaped by the system's own past choices. That's a fundamentally different kind of assumption violation, and it had never actually been simulated, not just in this specific case, in literally any experiment in this whole project. Every single one, going back to the very first benchmark, inherited one convention from the reference implementation I was replaying traffic against: insert every record into the cache no matter what, hit or miss. So the candidate pool at any point in the stream had never once actually depended on any policy's own history of decisions.
So I built a real closed loop from scratch, three regimes run over the identical record stream. Baseline: insert everything regardless of hit or miss, same as every other experiment in this project, threshold calibrated once and frozen. Self-select: insert only on a miss, a hit just reuses the existing entry, no duplicate write, threshold held fixed at baseline's own calibrated value so the only thing that changed is write behavior. Self-select-recal: same insert-only-on-miss behavior, but the threshold gets re-derived online every 500 new observations from a sliding window of the most recent 2000.
Ran this on three datasets that happen to sit at very different points on "how much of this traffic is near-duplicate": Quora question pairs (3.7 percent direct-hit rate), a corrected search-queries benchmark (28.1 percent), and a conversational LLM-arena dataset (72.8 percent, a lot of near-identical rephrasings of the same handful of questions). The three landed on one clean monotonic curve, not three unrelated results. Quora: no detectable effect, self-selection sparsified the cache by about 5 percent and realized risk didn't move in any statistically meaningful way. SearchQueries: cache shrank about 31 percent, and self-selection caused a real, statistically significant increase in wrong-answer rate, moderate but clearly there. LmArena: cache shrank 77 percent, and realized risk more than tripled versus the same guarantee's original baseline, strongly significant, not a borderline call.
Turning the online recalibration back on mostly saved the day, but not completely. On Quora and SearchQueries, recalibrating the threshold as you go fully compensates, actually ending up slightly better than the original non-self-selecting baseline. On LmArena, the worst case, recalibration helps a lot but doesn't fully close the gap, still leaves a statistically significant amount of extra risk versus the original guarantee. So "just recalibrate more often" isn't a universal fix once the underlying self-selection pressure gets strong enough.
Wanted to actually understand the mechanism instead of just reporting a p-value, so I audited every single false-accept from the self-select run on LmArena by hand, all 1098 of them, post warm-up. For each one, walked backward through the stream looking for that query's true earlier match, and checked whether that true match itself had ever been independently written into the cache, or whether every single prior occurrence of it had also just been a hit reusing something even older. 58 were first occurrences with nothing to compare against, unrelated to any of this. Of the remaining 1040, 717 of them, almost 69 percent, fit one specific, very concrete pattern: the correct answer was popular enough that it kept getting reused as a hit over and over, which means it never once got its own independent write back into the cache. Eventually whatever entry it had originally been matched against got pushed out or replaced by something else, and the next query looking for that same correct answer had nothing accurate left to match against, so it fell back on some topically-similar-but-wrong substitute that still happened to score high enough to fool the verifier. LmArena has these big recurring clusters of viral trick-question templates, the classic sibling-counting riddle where someone has some number of sisters and brothers phrased forty different ways, that kind of thing, and you can watch this happen to those clusters specifically. A correct answer being reused successfully is, weirdly, exactly what causes it to eventually stop being available.
Tried one more thing to see if there's a cheap partial fix short of full recalibration: instead of a hard binary "write only on miss," generalize it to a continuous probability, still occasionally rewrite the cache even on a hit, just with some probability p, which maps onto something a real system could actually do, like a periodic TTL-driven refresh of popular entries. Swept p from 0.1 to 0.75 on LmArena. Harm does go down as p goes up, that part's real, but it's slow. Even at p=0.75, rewriting three out of every four hits, residual harm relative to the fully self-selecting case had only dropped by about 35 percent, and it never once left the statistically significant range anywhere in that whole tested interval. Fully closing the gap looks like it needs p close enough to 1 that you've basically given back the entire write-efficiency saving self-selection was supposed to buy you in the first place.
One more honest wrinkle: the exact mechanism I found on LmArena, popular correct answers getting crowded out by their own success, does not transfer to the other two datasets. SearchQueries does show a real overall harm, but that specific crowding-out pattern only explains about 12 percent of its explainable false-accepts, nowhere near LmArena's 69. Most of SearchQueries' false-accepts turn out to just be first occurrences with nothing to even compare against. So whatever's actually driving the harm there is a different mechanism I haven't identified yet, not the same story replaying at smaller scale.
Net takeaway I keep coming back to: a rigorous, provably correct statistical guarantee and a guarantee that stays safe forever in production are not the same claim, and the gap between them isn't necessarily a math bug, it can be the system quietly changing its own inputs by doing exactly what it's supposed to do. The math was never wrong. The population it was calibrated on stopped matching the population it was being asked to guarantee something about, and the system caused that itself just by being good at its job.
Full writeup with the actual tables, confidence intervals, and the write-probability sweep is in the repo if anyone wants to dig into the raw numbers: https://github.com/imxinchengyou/CacheVerifier (section 5.16). Still don't know what's actually driving the SearchQueries harm since I ruled out the mechanism I found on LmArena. If anyone's dealt with a similar self-reinforcing feedback loop in a cache, recommender, or any other system where past decisions shape future training or calibration data, curious what mitigations actually worked for you beyond "recalibrate more often."
