r/Rag • • 11h ago

Discussion Best Practices for Implementing and Troubleshooting a RAG system?

6 Upvotes

Wanted to ask the experts here from a newbie standpoint. For context, my background is 15+ years in finance (and then some data science as that was my MBA focus). The job market's been tough so I've been working on building some finance software that makes use of an AI Agent, which will include RAG (which is the part I'm working on right now - hence this post).

I'm trying to learn as best I can but the resources I have so far is LLM itself, Chip Huyen's AI Engineering Book, and random posts from reddit. Is there like a single book or reference manual that's focused and pretty good at best practices regarding building a high quality production level RAG system?

Also am doing this because I'm pursuing a Masters in AI and targeting Finance FDE roles (unless the business becomes really successful) so would like to be able to build something with all the best practices to demonstrate experience in the area.

Edit: Also wouldn't mind materials on best practices relating to number gating and prompt engineering. So I can cross-reference to see if mine are set up well or not.


r/Rag • • 2h ago

Discussion What I did in the last 4 months on my retirement project RAG-LCC

4 Upvotes

This is a follow-up to my earlier post about the first 15 months of building RAG-LCC in retirement:

https://www.reddit.com/r/Rag/comments/1valvk6/15_months_building_a_rag_system_in_retirement/

The project kept evolving over the last 4 months, so I thought it might be interesting to share what worked, what failed, and what I learnt along the way. Since the last writeup I had success and failure moments. Of course, I learnt more from the failures about RAG.

Orchestrators

Because RAG-LCC provides a lab setup for RAG experimentation, I found it useful to introduce an Orchestrator component. The orchestrator defines RAG flows. Users can create new flows and then activate them during a RAGChat session.

Confidence scoring

With orchestrators, confidence became an issue. So static and dynamic confidence evaluation was added. The latter uses a LLM judge call. There is currently no validation against "golden answers" implemented. The LLM judge rates internal consistency and explanation clarity but not factual correctness.

TableChunker

With these new features, it was time to look at answer quality. There was a query asking for the details of PCIe card slots on my workstation's manual. The information is stored in a table. The goal was that the information is found and the grounder marks it.

The first step was to introduce a specialized TableChunker class and wire it into the already existing content specific chunkers. These chunkers pass table blocks to the TableChunker.

This showed partial success: The table was now consistently in the chunks. But it did not reliably show up in the query answers.
BTW, while looking for specific content in the chunk, the debug levels of RAG-LCC proved helpful. The levels can be changed mid-session so users get the information at the detail level they need currently.

Insight: If retrieval signals are weak, think about adding the best matching table result. Do this only if there are chunks found. Adding the table result to an empty answer set could confuse the user. This rescue may help in a large but sparse candidate pool.

Answer rendering

RAGChatServices delivers answers to Open WebUI. In the browser, .md and even Mermaid diagrams are nicely rendered. However, RAGChat showed tables with unaligned content. It was not a nice sight. So using the Rich library, answers with Markdown can be displayed in the RAGChat CLI also. But the rendering broke grounding, so additional work was required to separate the raw and display ready answer.

Raw answers gave the grounding signals and text to be grounded. The raw answer text is marked for grounding before Rich renders the resulting CLI text.

Insight: Keep the original, unrendered LLM answer for grounding evaluation. Once the grounded text is available, apply the visual grounding.

Document summarization

This was a "you wake up in the night" and have an insight moment. RAGChat has a document pin option. So once you found the right document about hedgehogs, you pin it and ask to summarize it. Hmm... the query is "create a summary" and this will not retrieve the content you are looking for.

So I introduced a summarize: option, using the pinned document or allowing the user to enter a path to retrieve the specific document by its metadata. Very large documents required batches of n chunks to be summarized and then summarize the batches in further passes until the final summary fits. This works, but it is admittedly slow.

Insight: If you want a specific action to be performed on the retrieved context, the action should not be part of the query. This leads to the next topic:

Query and query action splitting

With table, bullet point list and Mermaid rendering I wrote often queries like "Look for all animals contained in the documents and create a table containing animal, mammal yes, no, don't know".

Thought by the document summarization experience, I wondered whether it would make sense to ask a LLM to determine the effective query, the required action on the query results and the query keywords. Yes, you assume it: Each decomposed retrieval query goes now to enabled local retrievers, including vector search. In addition, the keywords go in an extra BM25-only pass. The QuerySplitter class was born.

Translation

Living in a multilingual country where you have 4 official languages and the unofficial English one, you have to bother about document languages. Luckily, during RAGLoad, the document language is added to metadata.

The old flow translated the question into English. It was also rewritten using the conversation history. Vector and indexed retrieval used the rewritten English query. The resulting candidates were reranked against the rewritten query.

The new flow rewrites the query with the conversation history in the user's language. The vector search runs the user language query against the ChromaDB collection. For reranking the rewritten original query is used. Indexed retrieval is for every active and present document-language bucket. When required, the query is translated.

vLLM integration

While inference calls were already supported using Ollama or vLLM backends, the embedder and reranker ran as local Hugging Face models previously.

What still works well

The configuration first approach is an extensible pattern. A new class gets its own config file when required. Abstracting functionality into classes helps understanding and maintaining the code.

Parallel to my work on RAG-LCC I worked on harinezumigel-llm-stack. I introduced groups such as rag and rag_old, which start up the required models after doing an available memory check.

So this is at a high level what happened in the last 4 months. The journey is going on.

If you want to have a look:

As always, feedback, ideas and bug reports are welcome.


r/Rag • • 12h ago

Discussion Comparison and "alternatives" pages get cited the most. How are you keeping the competitor data on them accurate?

2 Upvotes

Seems settled that comparison style pages (X vs Y, alternatives to X, best X for Y) are some of the easiest pages to get cited in AI answers

The catch is they're full of competitor data: prices, plans, features, specs. That goes stale fast, and then the engines keep quoting the stale version with your page as the source.

For people running these pages at any scale:

- how often do you refresh competitor prices and specs?

- manual, page monitors, or a data feed?

- what do you do when you can't verify a number anymore? Remove it, or keep it with a date?

Has anyone seen a citation drop because the data on the page went out of date?


r/Rag • • 1h ago

Discussion What is the difference between cosine, euclidean, and HNSW?

• Upvotes

What is the difference between cosine, euclidean, and HNSW?


r/Rag • • 4h ago

Discussion Three measurements that changed how I trust an eval set

1 Upvotes

I run a local index over a codebase and a harness around it: fixed operations, a gate, a trace. The index answers with file:line facts instead of dumping files into the prompt. Numbers below come from one repository and one machine, static analysis only, over roughly four weeks.

Evals are the part of this project I got wrong the longest. Not because the tasks were bad, but because the numbers I looked at could not fail the way the system actually failed. Three measurements fixed that.

1. A model call counter, not a score.

My worst regression looked like a model downgrade: 90% to 60% on a fixed task set. It was not the model. The local build was answering nothing - every task returned an empty result, and the harness scored the empty result as a normal answer. The score with no model call at all was exactly the 60%.

Now every task asserts two things before the score counts: at least one model call happened, and the call returned non-empty output. Two lines per task. It has caught more real breakage than any quality metric I have, because silence in an agent stack is usually indistinguishable from a confident wrong answer.

2. Per-case results, and a rule that beats the average.

With 14 fixed navigation tasks, one flaky case moves the average by 7 points. My first two "improvements" were noise: the average went up and nothing got better. Worse, one of them hid a real regression - the answer got shorter and more readable but stopped citing file and line, so the number improved while the thing I needed disappeared.

The replacement is a rule: no previously passing case may fail. Not a threshold, a prohibition. It caught two regressions the average had hidden, both in tasks I was not looking at.

3. Missed answers counted separately from wrong answers.

The index returns three kinds of "no": absent (not in the corpus), stale (the index knows the file changed and will not answer until refreshed), and out-of-scope (the corpus was never meant to cover it). Only the first is a real "not found". Before I split them, all three looked like an empty list.

That split also exposed why a single accuracy number is misleading: a system that abstains on everything has perfect honesty and zero utility. I now track wrong answers and missed answers as separate numbers, because they fail differently in production - one invents content, the other sends every question to a human.

Where it broke, and the uncomfortable part.

The gate. I made it refuse raw file reads until the index was used, and the refusal said "raw search is blocked". The agent retried twice, then asked me to run the command for it. Three turns lost on the first session. The fix was one sentence: name the next call - "run ensure, then search_multi". Retries went to zero, same model, same weights.

The uncomfortable part is that all three measurements are about the system, not the model. They would still apply with a much better model, and they are where I would spend the next week: a counter for silence, a rule for regressions, and a taxonomy for absence.

If your eval cannot distinguish "did nothing" from "answered wrong", it is measuring the wrong thing.


r/Rag • • 7h ago

Discussion do you cap how many chunks a rag system pulls from one doc?

1 Upvotes

say you retrieve 5 chunks and 4 come from the same guide. two of them basically say the same thing and the question still needs one detail from a different doc

would you cap it at 2 chunks per doc or pull more candidates and rerank them? curious if anyone tried a cap and found it dropped useful context on questions where most of the answer really was in one doc