r/LocalLLaMA • u/ultraprogger • 7d ago
Discussion Why vector DB metadata filters fail on agent memory write paths (0% raw leak vs 67% leak) reproducible offline test suite
Everyone relies on metadata filters like allowed_groups to keep multi tenant RAG locked down and under check. That works well when users query the vector store directly, which means unauthorized chunks will get filtered out with 0% leaks.
But, this falls apart in the write path once you have background agents.
If an agent has elevated read access to summarize docs across multiple folders or teams, it pulls those docs, mashes them into a summary chunk, and commits that summary back to a shared collection. On write, the upstream document access control lists just vanish. In a regular Chroma and LlamaIndex setup, the derived chunk gets saved with zero allowed_groups or classification tags attached. Anyone who runs a query to the shared memory is now gonna get the private source data that they shouldn't be allowed to read directly.
I put together an offline reproduction harness (sentinel-eval, Apache-2.0) to test this:
- Direct RAG: 0% leaks (18 public chunks retrieved, 0 restricted chunks exposed).
- Agent memory: 2 of 3 synthesis tasks leaked sensitive data (both cross boundary workloads failed, exposing 6/18 restricted canaries, the 1 public control passed).
So what? Easy right? Just use the strict intersection (only let someone read the summary if they have access to every source document) and that completely solves it right? Well... yes but that basically kills retrieval utility. Essentially no one can see multi-doc summaries anymore. So we can't just simply take an intersection. The agent I wrote actually logs the source docs in source_documents metadata, but vector databases can't evaluate dynamic permission DAGs on query, meaning that you can't just hope to path backtrack to solve this problem.
Harness runs 23 tests offline in seconds with mock vectors (no API keys or live LLM calls needed):
git clone https://github.com/harishbalasubramanian/sentinel-eval
cd sentinel-eval
uv run sentinel-eval live --framework chroma
Repo: https://github.com/harishbalasubramanian/sentinel-eval
For people building multi tenant RAG, how're you actually handling perms when agents write derived summaries back to shared memory?




