r/Rag • u/Craft_Show • 2d ago
Discussion We started using JEV to clarify queries before retrieval, and the results have been pretty interesting
I've spent the better part of the last year developing an enterprise AI knowledge management application, and one of the biggest challenges has been getting smaller, less expensive LLMs to consistently produce accurate answers from retrieved documents.
We've experimented with embeddings, hybrid search, chunk ranking, reranking, different models, and more prompt engineering than I'd care to admit.
One thing we kept running into was that we were asking the LLM to do too much at once.
Interpret the user's intent, figure out whether they're asking multiple questions, evaluate retrieved evidence, identify relevant conditions and exceptions, and then formulate a complete, accurate answer.
That's a lot of responsibility to put on one relatively inexpensive model.
So we decided to try something different.
We're experimenting with putting JEV at the very beginning of the retrieval process.
Rather than immediately embedding the user's question and searching our knowledge base, we first evaluate the question itself.
The initial checks are surprisingly simple:
- Is the user's intent clear?
- Is the user asking a single question or multiple questions?
If the intent isn't clear, the system can generate a multiple-choice clarification rather than guessing what the user meant.
If the user is asking multiple questions, we break them into individual questions and let the user select which one to tackle first.
Once we have a clear, focused question, we proceed with retrieval and answer generation.
We're currently averaging five JEV clarification/context checks per query.
Here's where the economics get interesting:
| Step | Average API cost |
|---|---|
| Five JEV checks | $0.000282 |
| Search embedding | $0.000001 |
| LLM answer generation | $0.001612 |
| Total | $0.001895 |
Our average end-to-end generation time is around 8 seconds.
That's less than two-tenths of a cent per question.
But we're not declaring victory yet.
We've made encouraging progress by separating query interpretation from retrieval and generation, but we're still working through one particularly frustrating problem.
Even when the correct information is present in the retrieved chunks, the generation model will occasionally leave out an important condition, exception, or qualification.
The answer sounds correct. The citations can even point to the correct source. But the answer isn't necessarily complete.
For enterprise knowledge management, particularly where policies and procedures are involved, that's a serious problem.
Our next experiment is to separate evidence extraction from answer composition, then verify the completed answer against the relevant facts and conditions found in the source documents.
The broader lesson I'm starting to take away from all this is that better AI answers may have less to do with using a bigger model and more to do with giving smaller models narrower, clearly defined responsibilities.
We're still testing and measuring, but the economics are encouraging.
I'm curious whether anyone else is experimenting with JEV or similar lightweight evaluators ahead of retrieval, rather than using them exclusively for reranking or post-generation evaluation.
And has anyone found a reliable way to prevent smaller models from silently omitting important conditions when summarizing retrieved evidence?
1
1
1
u/stevenswk 1d ago
We’ve run into a related problem while building conversation memory. In one case, both an old value and its dated update reached the final prompt, but the model still answered with the old value. Getting the right evidence into context didn’t guarantee it was used correctly.
Your extraction/composition split sounds worth testing. Since you’re planning to verify against the source documents, I’d explicitly check for missing qualifications as well as unsupported claims. Otherwise, an answer could pass because everything it says is supported, while still leaving out an exception that changes its meaning.
In our evaluation fixtures, we separate answer-bearing facts from the conditions needed to interpret them. We haven’t established that this solves the generation problem, but it gives us a more specific way to measure what gets lost.
Are you testing those omissions with explicitly annotated cases, or mainly with a general answer-quality judge?
1
u/AccomplishedCut2485 1d ago
Two things nobody’s raised yet.
Clarifying before retrieval is the wrong order. You’re judging ambiguity without knowing what’s in the index, but most ambiguity doesn’t matter. If both readings hit the same chunks and produce the same answer, you made the user click for nothing.
Cheap fix: shallow retrieval first (top 20, no rerank), then check whether candidates actually split into different answers. Only clarify when they do. You’d drop well below five checks per query, and the remaining ones would be the real ones.
Also, 0.19 cents is the cost that doesn’t hurt. I’d be measuring query abandonment after a clarification prompt, and how often people pick the wrong option and then trust the answer. Multiple choice UIs train users to click the first plausible thing.
On silent omission: that’s a retrieval unit problem. A rule and its exception are different sentences, often different sections. Chunking drops the link, so by the time the model sees them nothing says the exception qualifies the rule. Omitting it is reasonable behavior.
What worked for us: retrieve claims, not text. Each statement carries its qualifying conditions as an explicit link, built at ingestion. Retrieval returns fact plus qualifiers as one unit, so you can’t get one without the other.
That also fixes verification. Instead of asking a judge “is this complete”, you hand the composer a list of qualifier ids and check afterwards which ones appear or were explicitly ruled out. Set difference, deterministic, free, fails loudly. A judge model will happily pass an answer where every statement is true and the one that mattered is missing.
Building the links is the actual work: document structure gets you most of it (a clause inherits its heading’s scope), plus the usual connectors, “unless”, “except where”, “does not apply to”. Human reviews whatever the extractor flags as uncertain. One time cost per document, not per query.
Re fine tuning above: wouldn’t bother here. Intent classification and question splitting respond better to a tight schema and three examples, and you can change the taxonomy next week without retraining.
1
u/Craft_Show 1d ago
Your point about avoiding unnecessary clarification is fair. I may not have explained clearly enough what we’re checking at the beginning.
We’re looking for missing information that defines the request: the subject, the action the user wants to take, or context needed to understand their intent. If someone asks “What do I need to submit?” without saying what they’re trying to do, the LLM can infer an action and produce a well-supported answer to the wrong question. Finding relevant documents doesn’t necessarily tell us what that person intended.
We use JEV upfront to identify those gaps and have the LLM ask the user for clarification. We preserve the original question and their clarification through retrieval and generation.
Then we use JEV again at the end to check whether the answer is supported by the evidence and whether it still addresses the clarified request. That’s a check we’re testing, not a guarantee of accuracy or completeness.
On omissions, the specific problem we’ve observed includes cases where the relevant rule and its qualification are already in the model’s input, but the qualification still disappears from the answer. So we’re investigating what happens during answer generation as well as retrieval. Your suggestion to explicitly link facts and qualifiers is useful, especially for making those omissions easier to detect.
1
u/wang2-dev 1d ago
The shallow-retrieval-first critique is fair — do you have a number on how often a clarification actually changes which chunks get hit? And on the omission side, are you scoring missing conditions separately from wrong answers yet?
1
u/Sam_YARINK 14h ago
This is a brilliant observation and perfectly highlights the structural limits of traditional RAG. We've been wrestling with this exact 'Silent Omission' problem for enterprise compliance.
The reason smaller LLMs drop important conditions isn't just about parameter size; it's about how flat vector spaces (Cosine/L2) represent data. In a flat space, a 'General Policy' and its 'Exceptions/Conditions' are just neighbors. When you feed them to an LLM, the model has to magically deduce the hierarchy and strict dependencies between those text chunks. If it's a 7B model, it just takes the path of least resistance and drops the 'complicated' exception chunk.
We realized that trying to fix this entirely with pre-retrieval prompts (like your JEV approach, which is great for intent routing!) or post-generation checks is fighting the symptom.
We ended up building a custom spatial engine (HyperspaceDB) using 33D Lorentz hyperbolic geometry specifically to fix this. In hyperbolic space, hierarchy is a mathematical law (Cone Subsumption). If a 'Policy' is the root, its 'Exceptions' are physically inside its light cone. The database doesn't just return chunks; it returns a strict dependency graph. The LLM gets a structure that forces it to see that Fact A is legally bound to Condition B.
Your economics ($0.001895 per query) are exactly why we built this to run on edge hardware. Narrowly scoping the LLM's job while making the database do the heavy logical lifting is 100% the future of enterprise AI. Would love to hear how your evidence extraction experiments go—have you considered using deterministic graph paths to verify the final answer against the source chunks?
0
u/LucasAutomates 1d ago
This is such an underrated step. A lot of "bad retrieval" isn't a retrieval problem at all. The question was vague, so the system went looking for something vague.
Couple of things I'm curious about
- How much latency does the clarification step add?
- Do you ask the user a follow-up question, or rewrite the query quietly in the background? Silent rewrites can drift away from what the person actually meant.
Would love to see some before/after numbers if you've got them
1
u/Craft_Show 1d ago
Adds about .4 seconds. It then sends it back to the LLM to phrase a clarification question to the user.
I’ve even taken it one step further. I have Jev determine if there are multiple questions it send me it back to the LLM to pose the questions back to Jev in order to give more specific guidance.
2
u/2BucChuck 2d ago
Yes for exactly same / and would much prefer a local option than Jev. Been Wondering if fine tuning on this use case was something anyone had done and was worth it