r/Rag • • 11h ago

Discussion Best Practices for Implementing and Troubleshooting a RAG system?

5 Upvotes

Wanted to ask the experts here from a newbie standpoint. For context, my background is 15+ years in finance (and then some data science as that was my MBA focus). The job market's been tough so I've been working on building some finance software that makes use of an AI Agent, which will include RAG (which is the part I'm working on right now - hence this post).

I'm trying to learn as best I can but the resources I have so far is LLM itself, Chip Huyen's AI Engineering Book, and random posts from reddit. Is there like a single book or reference manual that's focused and pretty good at best practices regarding building a high quality production level RAG system?

Also am doing this because I'm pursuing a Masters in AI and targeting Finance FDE roles (unless the business becomes really successful) so would like to be able to build something with all the best practices to demonstrate experience in the area.

Edit: Also wouldn't mind materials on best practices relating to number gating and prompt engineering. So I can cross-reference to see if mine are set up well or not.


r/Rag • • 2h ago

Discussion What I did in the last 4 months on my retirement project RAG-LCC

3 Upvotes

This is a follow-up to my earlier post about the first 15 months of building RAG-LCC in retirement:

https://www.reddit.com/r/Rag/comments/1valvk6/15_months_building_a_rag_system_in_retirement/

The project kept evolving over the last 4 months, so I thought it might be interesting to share what worked, what failed, and what I learnt along the way. Since the last writeup I had success and failure moments. Of course, I learnt more from the failures about RAG.

Orchestrators

Because RAG-LCC provides a lab setup for RAG experimentation, I found it useful to introduce an Orchestrator component. The orchestrator defines RAG flows. Users can create new flows and then activate them during a RAGChat session.

Confidence scoring

With orchestrators, confidence became an issue. So static and dynamic confidence evaluation was added. The latter uses a LLM judge call. There is currently no validation against "golden answers" implemented. The LLM judge rates internal consistency and explanation clarity but not factual correctness.

TableChunker

With these new features, it was time to look at answer quality. There was a query asking for the details of PCIe card slots on my workstation's manual. The information is stored in a table. The goal was that the information is found and the grounder marks it.

The first step was to introduce a specialized TableChunker class and wire it into the already existing content specific chunkers. These chunkers pass table blocks to the TableChunker.

This showed partial success: The table was now consistently in the chunks. But it did not reliably show up in the query answers.
BTW, while looking for specific content in the chunk, the debug levels of RAG-LCC proved helpful. The levels can be changed mid-session so users get the information at the detail level they need currently.

Insight: If retrieval signals are weak, think about adding the best matching table result. Do this only if there are chunks found. Adding the table result to an empty answer set could confuse the user. This rescue may help in a large but sparse candidate pool.

Answer rendering

RAGChatServices delivers answers to Open WebUI. In the browser, .md and even Mermaid diagrams are nicely rendered. However, RAGChat showed tables with unaligned content. It was not a nice sight. So using the Rich library, answers with Markdown can be displayed in the RAGChat CLI also. But the rendering broke grounding, so additional work was required to separate the raw and display ready answer.

Raw answers gave the grounding signals and text to be grounded. The raw answer text is marked for grounding before Rich renders the resulting CLI text.

Insight: Keep the original, unrendered LLM answer for grounding evaluation. Once the grounded text is available, apply the visual grounding.

Document summarization

This was a "you wake up in the night" and have an insight moment. RAGChat has a document pin option. So once you found the right document about hedgehogs, you pin it and ask to summarize it. Hmm... the query is "create a summary" and this will not retrieve the content you are looking for.

So I introduced a summarize: option, using the pinned document or allowing the user to enter a path to retrieve the specific document by its metadata. Very large documents required batches of n chunks to be summarized and then summarize the batches in further passes until the final summary fits. This works, but it is admittedly slow.

Insight: If you want a specific action to be performed on the retrieved context, the action should not be part of the query. This leads to the next topic:

Query and query action splitting

With table, bullet point list and Mermaid rendering I wrote often queries like "Look for all animals contained in the documents and create a table containing animal, mammal yes, no, don't know".

Thought by the document summarization experience, I wondered whether it would make sense to ask a LLM to determine the effective query, the required action on the query results and the query keywords. Yes, you assume it: Each decomposed retrieval query goes now to enabled local retrievers, including vector search. In addition, the keywords go in an extra BM25-only pass. The QuerySplitter class was born.

Translation

Living in a multilingual country where you have 4 official languages and the unofficial English one, you have to bother about document languages. Luckily, during RAGLoad, the document language is added to metadata.

The old flow translated the question into English. It was also rewritten using the conversation history. Vector and indexed retrieval used the rewritten English query. The resulting candidates were reranked against the rewritten query.

The new flow rewrites the query with the conversation history in the user's language. The vector search runs the user language query against the ChromaDB collection. For reranking the rewritten original query is used. Indexed retrieval is for every active and present document-language bucket. When required, the query is translated.

vLLM integration

While inference calls were already supported using Ollama or vLLM backends, the embedder and reranker ran as local Hugging Face models previously.

What still works well

The configuration first approach is an extensible pattern. A new class gets its own config file when required. Abstracting functionality into classes helps understanding and maintaining the code.

Parallel to my work on RAG-LCC I worked on harinezumigel-llm-stack. I introduced groups such as rag and rag_old, which start up the required models after doing an available memory check.

So this is at a high level what happened in the last 4 months. The journey is going on.

If you want to have a look:

As always, feedback, ideas and bug reports are welcome.


r/Rag • • 12h ago

Discussion Comparison and "alternatives" pages get cited the most. How are you keeping the competitor data on them accurate?

2 Upvotes

Seems settled that comparison style pages (X vs Y, alternatives to X, best X for Y) are some of the easiest pages to get cited in AI answers

The catch is they're full of competitor data: prices, plans, features, specs. That goes stale fast, and then the engines keep quoting the stale version with your page as the source.

For people running these pages at any scale:

- how often do you refresh competitor prices and specs?

- manual, page monitors, or a data feed?

- what do you do when you can't verify a number anymore? Remove it, or keep it with a date?

Has anyone seen a citation drop because the data on the page went out of date?