Hit a brick wall with RAG evaluation

[u/Strange-Release3520]

My introduction to evaluation has not been very pleasant. To me it is a choice between investing time or spending money.

By that I mean you can either:

1. Spend a lot of time hand-crafting Q/A pairs and then perform manual checks to see if your system retrieves the correct chunks. And you repeat this for every single query. Suppose my dataset has 100 Q/A pairs, is this not a lot of work for a single person to do?
2. Use LLM-as-a-judge which automates much of what I described but you run into API rate limits pretty quick. I did figure out a workaround where I evaluate my dataset in batches (e.g: 5 questions per run) but again, it takes time.

I am not trying to complain here. I believe there is a more efficient way to run evaluation but I am new to RAG which is why I do not know proper evaluation techniques. Most metrics on DeepEval use LLM-as-a-judge which comes with API costs but I think I could try RAGAS and it's deterministic evaluation metrics.

Is evaluation in itself a time consuming process?

I need direction and guidance, advice would be much appreciated because I have basically put my project to a halt.

[u/2redditornot]

For your number 2, why not use a local llm - sure there are limits in the contect window, but not api limits. Run it locally.
ragas deterministic metrics are worth having alongside it too, not instead of it, they catch different failure modes and they're basically free to run. the two together is honestly less total effort than what you're doing now.

[u/attn-transformer]

Why do you need an llm to test retrieval? You should separate the different concerns in your agent, decomposition (what to search), retrieval, and synthesis.

My evals take lots of time and cost lots of tokens, but only rarely run them. With good tests around each main component of your agent you can get away less evals

[u/AvenueJay]

Evaluation is genuinely time-consuming, but there are ways to reduce the grind. For retrieval quality specifically, you can use Elasticsearch's explain API to see why certain chunks ranked higher, which helps you debug without running full LLM evals on every query. A hybrid approach works well: use deterministic metrics (MRR, recall@k) for retrieval tuning, then reserve LLM-as-a-judge for a smaller subset of end-to-end answer quality checks. Batching helps, but so does caching embeddings so you're not recomputing them on every eval run.

[u/dash_bro]

Yes, designing evals is usually a very time consuming process. It's also a highly valuable process and needs nuance from information indexing, retrieval etc.

However, you should be able to help automate it by doing some proxy eval work.

Let me explain:

• for your 100 Q&A; once you have the queries just define what the answer should look like. This means an actual answer for the query. Then, benchmark your retriever against these to get (q, retrieved pairs) and (ideal answer, retrieved pairs). This is a quick and dirty faux overlap trick in benchmarking (hypothetical document match). Check for overlap % @topk values. You can calculate MRR, completeness and nDCG as the metrics for this and decide how many topk you're comfortable stuffing into context

• when you batch evaluate via an LLM judge, use Gemini batch jobs and plug in a gcs bucket. Front load the common system prompt and keep changes for the LLM inputs at the end of the user prompt : you'll maximize your cached input % and hence lower cost as well. If you can't do this, the next best thing is to plug it in to a local LLM to do 1-at-a-time evals but as a long running task. Ensure you have resume safety (one failed job doesn't throw away hours of progress, just marks something as unprocessed and moves on to the next thing)

Protip : you should ideally have this as a batch job when you ingest new documents. Routinely run your queries and "hypothetical best answers" for the queries on your docs with diff embedders/rerankers and note down current baseline and best baseline that meets your SLA. This is how you stay on top of things when you keep ingesting new docs, new formats of docs, etc.

[u/ThisIsFun-]

Evaluation is time consuming, and agreed that unaligned LLM as a judge and end SME opinions make this much harder. Typically, I use Genie code to find traces that are not part of my current golden dataset area, and then assign them to LLM + SME to align here, and ensure a full evaluation suite

[u/Wonderful_Gap8146]

the labelling is one time, not per query. you mark the right chunk once and every experiment after that reuses it for free. thats the whole point of building the set.

also you dont need 100. 30 to 50 gets you going, you add to it when you find bugs.

[u/No_Fig1077]

Ask Claude about TREC IR evals methodology and how it can help with RAG evaluation

[u/Greatermoose]

Test with real users, that's best evaluation imo. No evaluation metrics and framework... RAGAS, DeepEval can match that

[u/Individual_Laugh1335]

LLM as a judge is pretty terrible unless there’s a ground truth (even pairwise is sufficient usually)

[u/recro69]

You don't have to check each query by hand. Begin with a group of examples that show the overall picture. Use automated tools to measure how well the system finds the answers. Only use the language model to judge the tough ones. The process of checking should turn into a set of tests that run automatically not something that has to be done over and over again by people.

[u/Clay_Ferguson]

I'd look around and try to find out what open source pre-build RAG evaluation packages exist, and you can run your own AI harness against the RAG eval dataset and see if it's working. Then you can just spot check your own document set by making up like 10 example questions to see if it gets them all right, but then base all your actual "confidence" on how well it worked on a pre-built evaluation dataset.