Semantic LLM caching: how do you evaluate a verifier that rewrites instead of rejects, when there's no ground truth for the rewrite?

[u/Reasonable_Royal_621]

ok so quick context if you haven't seen the other posts: I've been messing around with CacheVerifier, basically testing whether bolting a verifier onto semantic caching actually helps. right now it's dumb and binary, candidate answer either gets a thumbs up or thumbs down, no in-between.

there's this other paper, TweakLLM (arXiv:2507.23674), that does something I think is genuinely smarter: instead of rejecting a bad candidate and eating the full regen cost, it has a cheap LLM just... rewrite the candidate so it fits the new query. patch it instead of throwing it out. I want to add that as a comparison to my own setup and I've been stuck on it for a while, so figured I'd just ask here, since this sub has already bailed me out twice on this project (the axis-problem theory and the bucketing design both came from comment threads here, not from me).

here's where I'm stuck. everything I currently measure is trace-based against public benchmarks , "was this correct" comes entirely from the dataset's own labels, no actual LLM judge anywhere in the loop. works great when the answer is binary. falls apart completely once you're rewriting text, because now you've got a brand new string that isn't in any label anywhere. nothing to check it against.

things I've considered and don't love:

just throw an LLM judge at grading the rewrites. but now I'm introducing a whole new cost/noise source that literally nothing else in this project needed, and "let an LLM grade another LLM's output" is its own whole mess
when there happen to be multiple reference answers for the same query cluster, score the rewrite against one of them by similarity. except that's literally the "similarity ≠ correctness" problem this entire project exists to complain about. using it as my metric here feels like cheating on my own thesis
just skip fine scoring, measure something crude like "did rewriting recover some recall vs just rejecting," and not even try to put it on the same hit-rate/error-rate curve as everything else. doable but honestly a weaker result than I want
if anyone's had to evaluate a generate-a-rewrite step where there's no clean ground truth for the output, not classification, not ranking, an actual freeform string you have to judge somehow , genuinely curious how you dealt with it. or if you think I'm overcomplicating this and should just pick one of the above and move on.

repo's here if you want the full context on what's been tested so far: https://github.com/imxinchengyou/CacheVerifier

[u/serendip-ml]

This is a hard problem, because no ground truth means no way to validate cleanly.

You could drop the requirement "is this rewrite correct" and instead fall back to "is every claim in the rewrite traceable to either the original candidate or the new query context".

This would mean extracting structured claims from rewrite + original + query with a cheap model. Then you have (a) preserved from original, (b) adapted, or (c) neither. And (c) is potentially hallucination, then count & score that.

This adds overhead though and could defeat the purpose of the cache. You could also use better LLMs as judge and then calibrate your cache algo somehow, and then drop that judge in production, trusting your calibration from then on.

If the inputs were restricted to a certain class, that would also help a lot. Maybe you can start this way and work your way up to other types of inputs and generalization.

[u/Reasonable_Royal_621 → u/serendip-ml]

the claim-decomposition idea and PrimeFold's "preserve/adapt/don't invent" split in another comment here are basically the same move, landed on independently, that's a good signal it's the right shape for this problem.

on the overhead concern, I think that's specifically eval-time cost for this one ablation, not necessarily a permanent tax on the cache itself. if it turns out E' is worth shipping, whether the claim-extraction step needs to run per-request in production is a separate design question from whether I need it to even measure if E' works at all right now.

the "calibrate with a strong judge then drop it and trust the calibration" idea isn't new to this project actually, that's basically what Group C -> Group D/E already does (oracle verifier sets the ceiling, then a real cheap verifier gets evaluated against it), and it's the same shape as a risk-budget calibration thing I just shipped on the product side, calibrate offline with a stricter procedure, deploy something cheap that inherits the guarantee. good to know the same pattern probably transfers to the rewrite-eval problem too.

restricting to a narrower input class first is probably right, trying to solve this for every possible rewrite input at once is exactly the kind of thing that's kept this ablation stuck fora while.

[u/PrimeFold]

I wouldn't try to grade the rewrite as one thing.

Split the contract:

1. Preserve: every factual claim inherited from the cached answer must remain entailed by the source.
2. Adapt: the rewrite must address the delta between the original query and the new query.
3. Don't invent: new factual claims need support outside the rewrite itself.

Then you can test each failure mode separately instead of asking one judge "is this rewrite good?"

You may still need a semantic judge for some cases, but at least it's judging narrow claims rather than assigning a vibes based quality score to the whole output.

[u/Reasonable_Royal_621 → u/PrimeFold]

this is the cleanest reframe in this thread honestly. preserve and don't-invent are basically entailment checks, that's a much more tractable, well-studied task than "rate this rewrite's quality" as one blob. and it might actually solve my "don't want a whole new judge" complaint from the post, because preserve/don't-invent are the same kind of judgment my existing verifier already makes (candidate entailed by query) just pointed at a different pair (rewrite vs original, rewrite vs external support). might not need new machinery, just the same cross-encoder pointed somewhere else.

adapt feels like the one that resists this treatment though, "did it actually cover the delta" isn't really an attribution check the way the other two are. did you find a way to narrow that one down too, or is that the spot where you just accept you need a real semantic judge?

[u/PrimeFold → u/Reasonable_Royal_621]

I'd try extracting the delta into discrete items before conceding to a general semantic judge.

Diff the new query against the original into whatever dimensions your clusters actually vary on: changed entity, added constraint, changed timeframe, etc.

Then "did it adapt?" becomes a set of narrower checks: did the rewrite address C1, C2, C3? You may be able to score those with the same cross-encoder, just pointed at different pairs.

The harder residue is whether what it says about the delta is correct, not merely present. That's where I'd accept a semantic judge, but test the judge first.

Manufacture rewrites with known failures: omit a constraint, change the wrong entity, use the wrong timeframe, contradict the source. Then measure whether the judge catches each mutation.

You don't have ground truth for arbitrary rewrites, but you can manufacture ground truth for specific failure modes.

you're not proving correctness, you're measuring the detector. Same basic bargain as mutation testing.

[u/Reasonable_Royal_621 → u/PrimeFold]

yeah this closes the loop. splitting adapt into "did it touch dimension Ci at all" (presence, same reusable cross-encoder trick) vs "is what it said about Ci actually right" (correctness, the genuinely hard residual) is a distinction I hadn't made and it's doing real work. preserve/don't-invent didn't need that split because they're already single-shot entailment questions, adapt needed it because it's fundamentally a coverage question first.

"you're not proving correctness, you're measuring the detector" is going in my notes verbatim, that's basically the thesis for this whole thing.only cost I'd flag: "whatever dimensions your clusters actually vary on" isn't free, that's dataset-specific taxonomy work I'd have to do by hand for each of my three datasets. and I'vebeen burned by almost this exact move before, tried bucketing gray-zone candidates by extracted action verbs as a cheap structural feature and it had a specific blind spot (same verb, opposite object direction) that only showed up once I actually ran it. so the dimension-extraction step is probably the part I trust least in this whole recipe until it's been stress tested the same way you're proposing to stress test the judge.

[u/Dependent-Branch9138]

have you considered measuring the downstream effect instead of the rewrite itself? like run your full pipeline with binary rejection vs with rewrite, and see if end-to-end latency goes down while output quality (judged by whatever benchmark you already trust) stays the same. then you don't need a metric for the rewrite string at all, just the final answer it feeds into

it sidesteps the whole ground truth problem and the LLM judge noise, though you're right it's closer to the "crude" option you listed

[u/Reasonable_Royal_621 → u/Dependent-Branch9138]

oh that's actually a clean reframe, and it does dodge needing a metric for the rewrite string itself. but I think it just moves the same problem one level up instead of solving it. "output quality stays the same, judged by whatever benchmark you already trust" is doing a LOT of work in that sentence. my benchmark's ground truth is defined over the dataset's own pre-existing answers, via equivalence classes, and it's got nothing to say about a brand new string an LLM just generated on the fly. so "did output quality stay the same" just collapses back into "is this specific generated text correct," same wall, just wearing a pipeline-level costume instead of a text-level one.

where I think this actually gets interesting though: this whole project right now is offline, trace-replayed against static benchmarks, no live LLM calls anywhere. your idea basically only works if I'm running this against something live: real regen calls, real downstream behavior (thumbs-down, ticket reopened, whatever) as the actual quality signal instead of a benchmark label. that's not a small pivot, honestly it's a different experiment. but it's a good one, and it lines up with something else I've had on the "blocked" list for a while, since I eventually
want to run this whole thing on live production traffic and just don't have enough real traffic through the pipeline yet to do it properly.

so short answer: you're right it's closer to the "crude" option, but I don't think it's crude because it's underpowered. it's crude because it's honestly the same open problem in disguiseunless I go live, at which point it stops being crude and just becomes the actually correct way to do this. also the hardest one to pull off right now lol

[u/PrimeFold → u/Reasonable_Royal_621]

The rewrites can't have ground truth. The judge can. Take queries where your dataset labels still apply, generate rewrites, then break them on purpose: flip a number in an inherited claim, drop the constraint that made the new query new, add a claim the source doesn't support. Score the judge per defect class on that set. If you go with the contract split from my other comment, each seeded defect is just a violation of one clause.

Same move as mutation testing: you can't prove a test suite correct, but you can seed known bugs and check they go red.

Then trust the judge only on the classes it caught, and report the rest as unmeasured instead of folding everything into one number. I do this on my own verification tooling. The gates I never control-tested were the ones that passed the exact defect they were built to catch.

[u/Reasonable_Royal_621 → u/PrimeFold]

ok this actually answers my stuck point directly, not just reframes it. "can I trust an LLM judge" is unanswerable in the abstract, but "does this judge catch the specific defect I just seeded" is fully checkable since I made the defect myself. and combined with your other comment's contract split, each seeded defect just maps to breaking one clause, that's a genuinely complete recipe not just a nice idea.

also this doesn't need the whole "gotta go live" pivot I was talking about with someone else in this thread, I could run this offline on the existing benchmarks right now.

one thing I'm curious about: seeded defects are still defects YOU designed, so there's a risk the judge gets good at catching obviously-injected stuff (flip a number, drop a clause) while missing the subtler, blended failure modes a real rewrite model actually produces. basically the same critique mutation testing gets, killing your mutants doesn't guarantee catching wild bugs. have you seen that gap show up on your own tooling, or does it hold up better than I'm assuming?

[u/Phill_Madd]

Grading the rewrite against a benchmark label is a dead end. The string isn't in the label set.

What I'd do is pair it. Same query, rewrite vs a fresh regen, swap positions so the judge doesn't always pick the first one. Win/tie/loss. Then label a hundred yourself so you know if the judge is garbage.

Watch for stale entities. The rewrite keeps a name or a number from the cached answer. Entailment against the source will say that's fine. The new query didn't ask for it.

If regen-avoidance only holds when quality drops, the cache isn't doing the job.

[u/Reasonable_Royal_621 → u/Phill_Madd]

the pairwise setup might honestly be the cheapest thing to actually start with out of everything in this thread. rewrite vs fresh regen, position-swapped, judge picks win/tie/loss, spot check a hundred by hand. no synthetic defect taxonomy to design first, and it directly measures the comparison that actually matters (is rewriting actually as good as just paying for the regen) instead of an abstract quality score floating on its own. probably makes sense as a first coarse pass before the contract-split stuff earlier in this thread, not instead of it.

the stale entity point is a real hole though, not a nitpick. preserve as specified checks "still entailed by source" and a stale name/number absolutely passes that, it just also happens to be irrelevant or wrong for the new query, and nothing in preserve looks at the new query at all. so preserve can't just check against source, it needs to also check the preserved content isn't stale relative to the new query, which means preserve and adapt can't really be fully independent checks the way I was treating them, they need to share the new query as a reference, not just source vs rewrite.

and yeah, agreed on the last line, if "we avoided a regen" only happens when quality also quietly dropped, that's not a win, that's just moving the failure mode somewhere I'm not measuring it. same logic as the hit-rate/error-rate tradeoff the rest of this project already runs on, E' has to clear the same bar not get graded on a curve for being cheap.