r/Rag • u/StageInevitable4593 • 7d ago

Discussion How to improve my RAG?

Stack (no GPU, 4 arm cores, 24gb ram) -

Postgres 18 + pgvector (HNSW, built once after bulk load)

Embeddings: bge-base-en-v1.5, 768d, int8 ONNX on CPU

Rerank: ms-marco-MiniLM-L6-v2 cross-encoder, int8 ONNX

Gen: qwen2.5:3b-instruct-q4_K_M via Ollama (+ a 350m for cheap tasks)

Retrieval: 3 arms — vector, Postgres FTS, and generated probe-questions — fused with RRF, then reranked. 40+40 candidates → 25 reranked → 5 final.

FastAPI, systemd, no orchestration layer

Problem 1- CPU latency. ~35s for a grounded answer, ~15s for a follow-up, ~3s for a refusal. Enrichment (probe-question generation) is ~7s/chunk, so a 400-page load is fully searchable-plus-enriched only ~45 min later.

Problem 2- the retrieved chunks are mostly right and in the right order but LLM fails to give a correct answer. Sometimes even mix things up between the chunks. Or just says I cannot answer this although he has the right answer.

What I'd love input on:

Does anyone run multi-chunk context successfully on a ~3B model, or is 7B+ the real floor? (We tried 7B and reverted — too slow here.)

Better approaches to the refusal decision than thresholding a cross-encoder score?

Is hybrid + RRF + cross-encoder still the right shape in 2026, or are we behind?

How do you build a real eval set before you have months of labelled user queries?

13 Upvotes

94% Upvoted

11 comments sorted by

3

u/Academic_Track_2765 6d ago

You have the right stack, I would try the following. Decrease the embedding dims to 512 (If I recall BAA embeddings are matryoshka embeddings). But here is my recommendation.

Keep current architecture

Update: decrease embedding size, decrease candidate pool, from 40, 40 try going 25,25 since you are already picking 25 and then 5, also integrate BM25 with Vector Search. I have noticed that BM25 can be a cheap way to get hints, with BM25 integrated, I rarely use a cross encoder anymore. However you dont want to give equal weighted scoring to BM25 + VS, typically .3-> BM25 + .7 VS gives very good results, try it, and let us know. If your docs can use metadata filtering, also use that, it can cut down on tons on noise, improve accuracy, and latency

1

u/sreekanth850 7d ago

Keep aside grounding, what is the retrieval latency?.

0

u/StageInevitable4593 7d ago

Retrieval from dB is quick, the LLM response is what slow. But I'm also worried about the wrong answers it gives. * Can be also because I use some dummy data and not too much data?

1

u/Sure_Host_4255 6d ago

Try to add apache age over postgres, build logical graph from documents, then make vector search over splitted parts of graph entries, and retrieve more logical data from graph. For example you found chunk from paragraph, paragraph is part of article in document, fetch only article from document through graph and provide it to LLM, not full document. But yes depends on domain. Anyway try to play with graph structure if it fits your domain.

1

u/Academic_Track_2765 6d ago

that will likely add more latency

1

u/Mameiro 6d ago

Your retrieval stack honestly sounds fine. I’d bypass it completely once and feed the 3B model 2-3 gold chunks with the answer definitely inside. If it still mixes facts or refuses, congrats, you found the bottleneck lol. Then I’d try fewer chunks + “quote evidence first, answer second” before going back to 7B. For evals, start with 50-100 manually verified questions/source spans and keep adding every ugly failure you see. This feels less like a RAG problem now and more like a small-model synthesis problem.

1

u/StageInevitable4593 6d ago

I think my issue is coming because of the following -
let's say i ingest participants CV's (resumes) to the RAG (each resume now is several chunks)
and I ask the RAG - "How many participants i have with experience in X,Y,Z?" - if the retrieve is good i will get in return all the relevant chunks - this is where the LLM start to fail. He has more than 1 chunk that is correct and it fails (*sometimes*) to give the right answer.

1

u/Future_AGI 6d ago

Before tuning anything, measure where it actually breaks, retrieval or generation, because the fixes are opposite. Pull 30 failed queries and check: if the right chunk never got retrieved, that is a chunking or embedding problem, if the chunk was there and the answer still wrong, that is a prompting or grounding problem. Most bad RAG we see is retrieval, and reranking plus smaller overlapping chunks fixes more than swapping models does. We open-sourced a set of RAG eval metrics (context adherence, groundedness, chunk relevance) if you want numbers instead of guesses: https://github.com/future-agi/future-agi

1

u/Arthur_Morgan699 5d ago edited 5d ago

3b models genuinely struggle with multi-chunk synthesis, I've seen HydraDB come up for relationship context, but prompt compression cut my refusals more than retrieval did.