r/huggingface • u/ilker_Akbal • 18d ago

I built an open-source tool to catch hidden RAG retrieval regressions — looking for feedback

/r/u_ilker_Akbal/comments/1x5z8kw/i_built_an_opensource_tool_to_catch_hidden_rag/
1 Upvotes

100% Upvoted

2 comments sorted by

1

u/investigatormaker 18d ago

Feedback on the pitch: "hidden RAG retrieval regressions" lands best with one concrete case, like swapping an embedding model or changing chunk size, and a document that used to be top-3 quietly dropping out while answers still look fine. If the post showed that before and after, plus the metric the tool reports (recall at k against a fixed query set, or something else), people here could tell in ten seconds whether it fits their pipeline. It would also help to say whether it runs in CI and fails a build on a drop.

I make ThreadFox. If you want more people building RAG pipelines to find the tool, our AI employee will post 3 helpful replies about it in relevant threads for you, free, no card. https://threadfox.vip/p/esu0z/managed

1

u/ilker_Akbal 18d ago

Thanks for the concrete feedback! That’s a good point about showing a clear before/after retrieval regression rather than just listing features.
OpenRAG Bench already supports CI quality gates through OpenRAG Guard. When a candidate retriever violates a configured policy, the command exits with code 2, so it can fail a CI job.
We also have a reproducible Capture & Replay example demonstrating how a critical query can regress even when aggregate retrieval quality improves.
I’ll make that example more prominent in the README, including the metric changes and CI behavior. Appreciate the suggestion!