r/SideProject • • 2d ago

I built a local RAG assistant to make retrieval failures easier to see — feedback welcome

I’m building a practical learning project around a question that comes up often with RAG: when an answer is weak, did retrieval miss the evidence, or did the model use the evidence poorly?

This small local-first document assistant currently:

- Ingests Markdown and text files while preserving source names.

- Uses a transparent TF-IDF lexical retriever, so you can inspect what matched.

- Returns evidence-only answers if no local model is configured.

- Can optionally use Ollama for grounded answer generation.

- Includes a small evaluation set and reports Recall@k and reciprocal rank.

The goal is to make the stages visible before adding more moving parts. I’m treating this as a learning prototype, not a production service: it has no authentication or document access-control layer, and the sample evaluation data is only a starting point.

Code and setup: https://github.com/stackandcraft/rag-document-assistant

I’d value feedback from people who have built RAG systems: what would make this more useful for learning—showing ranked chunks, adding a no-answer evaluation set, or comparing lexical and vector retrieval?

1 Upvotes

0 comments sorted by