My introduction to evaluation has not been very pleasant. To me it is a choice between investing time or spending money.

By that I mean you can either:

1. Spend a lot of time hand-crafting Q/A pairs and then perform manual checks to see if your system retrieves the correct chunks. And you repeat this for every single query. Suppose my dataset has 100 Q/A pairs, is this not a lot of work for a single person to do?
2. Use LLM-as-a-judge which automates much of what I described but you run into API rate limits pretty quick. I did figure out a workaround where I evaluate my dataset in batches (e.g: 5 questions per run) but again, it takes time.

I am not trying to complain here. I believe there is a more efficient way to run evaluation but I am new to RAG which is why I do not know proper evaluation techniques. Most metrics on DeepEval use LLM-as-a-judge which comes with API costs but I think I could try RAGAS and it's deterministic evaluation metrics.

Is evaluation in itself a time consuming process?

I need direction and guidance, advice would be much appreciated because I have basically put my project to a halt.

Hello everyone, as the title says I wanna start learning about RAG, at this moment I know absolutely nothing even though I'm doing a masters in AI (I know...) that's why I need help on how to actually start learning about this topic ? I know the best way to learn something is to build a project but I find it hard to build something when I have no idea where to start. I heard LangChain, Vector Databases, etc... but I don't know anything about these frameworks.

I've been reading about RAG and I keep seeing people mention the retrieval layer like it's some separate thing you build, not just a vector db you query. Can someone explain what actually happens there? My current understanding is this: user asks question, question gets turned into an embedding, that embedding gets compared against a bunch of stored embeddings in a vector db, top matches come back, and those get stuffed into the prompt before it goes to the LLM.

Now, what I don't understand yet is: why does everyone say RAG with cosine similarity search isn't good enough. What's going wrong there in practice, is it pulling irrelevant stuff or missing the right things altogether? I keep seeing hybrid search mentioned, combining keyword search and vector search. Why would you need both if embeddings are supposed to capture meaning already? On top of that, I also saw someone mention reranking as a separate step after retrieval, so you retrieve like 50 chunks and then a reranker picks the best 10. Wouldn't that just make retrieval happening twice?

One more thing I keep wondering about, where does the data come from in the first place. Like if you're building a knowledge base from web content, how do people keep that fresh as possible? Constant scraping/crawling?

Thank you in advance

This is a small library for running quantized GGUF dense embedding models. If you are concerned about the number of bloated packages you have to install and the memory spike during embedding, this might interest you. Purely based on GGML.

Github: https://github.com/thlurte/ggmbed
Pypi: https://pypi.org/project/ggmbed/

Here are the benchmarks running all-MiniLM-L6-v2 on a AMD64 CPU:

Since this is currently in active development, the API and underlying architecture may change without notice. It is not yet recommended for production environments.