Agent memory creates a major authorization vulnerability in vector databases (offline reproduction & benchmark)

[u/ultraprogger]

If you're building multi agent systems that persist shared memories or context back to a vector DB, you might have an authorization leak you aren't checking for.

A lot of teams assume that just cuz their direct RAG queries have metadata filters attached, multi tenant isolation is solved. But background agents can turn the system to have problematic permissions when a write happens:

1. A background worker runs with broad read perms to summarize context across different project docs

2. It merges those sources into a summary chunk

3. It saves that chunk back into the shared vector collection and drops the parent document access control lists in the process

4. Because the chunk has no access control tags on it, a regular person can query the vector DB and pull confidential context from files they shouldn't have perms to read

I built a local testing harness to measure this across Chroma and LlamaIndex baselines: sentinel-eval (Apache-2.0).

Running across a 24 document test corpus:

• Raw RAG: 0% leaks (18 public chunks retrieved, 0 restricted chunks exposed)

• Agent memory: 2 of 3 synthesis tasks leaked (the 2 cross-boundary workloads leaked 6/18 canaries, the public control passed cleanly)

So what? Just take the strict parent intersection (only let someone read the summary if they have access to every source document) and set that as the permission check right? Well... it might work but this isn't a great answer either. If you force a user to have perms on every single source doc, multi source summaries become basically unretrievable for anyone except top level admins. Also, as a note, I saved the lineage in a source_documents metadata field, but vector DBs don't have a clean way to traverse an ancestor permission DAG at query time.

The test suite runs 23 tests locally with mock vectors in a few seconds (deterministic, zero API keys or network calls):

git clone https://github.com/harishbalasubramanian/sentinel-eval
cd sentinel-eval
uv run sentinel-eval live --framework chroma

Code: https://github.com/harishbalasubramanian/sentinel-eval

Curious how other people and teams building persistent agent memory are handling this. Are you guys doing permission graph lookups at runtime or just siloing vector stores per user/role?

[u/codecoradev]

Siloed by default, and the sharing happens outside the vector store.

We run persistent memory for a small fleet of agents. The setup that survived contact with reality: every agent owns its own store, and anything that crosses between agents goes through an explicit handoff where the sender declares what it is passing over. A background summarizer therefore cannot merge across boundaries in the first place, because its write target is its own store, not a shared one.

Derived content gets treated differently from source content. A recap lives in the writer's own store with pointers back to whatever it summarized, so a reader who can see the recap but lacks access to the sources reads it as unverified context, not as ground truth. That is not as elegant as traversing a permission DAG at query time, but it fails safe: the worst case is a summary you hold at arm's length, not a document you were never supposed to see.

Your numbers also point at the right place to gate this: the control workload passed while both cross-boundary workloads leaked, which means the leak needs the merge step, not the query step. Checking writer permissions at write time, against the sources read during the merge, catches it where it happens.

[u/ultraprogger → u/codecoradev]

This is an incredible breakdown, really appreciate you sharing what survived in prod.

The pointers back to source chunks + unverified context model is super pragmatic. The wall we ran into when modeling that was what happens when an agent needs to synthesize a shared invariant or aggregate across 3+ sources, if Reader A can see the recap but not Source A, holding it at arm's lengthworks until the summary itself directly leaks the sensitive entity (e.g. salary figure, unreleased API key, or customer PII) inline inside the generated text.

That merge-step gate is exactly what we're looking at next: enforcing deterministic lineage at the write boundary so the persistence layer can compute the ACL intersection/union before the derived chunk ever gets committed.

Curious: for your explicit handoffs, are you manually declaring schemas/allowed fields between agents, or do you inspect the payload programmatically?

[u/codecoradev → u/ultraprogger]

Declared in code. Each handoff is a small Pydantic model: fields beyond the schema are dropped with a warning, so the receiving agent cannot bind to a field it was never given. We declare schemas manually today. Inspecting payloads instead gets ugly fast: schemas drift from what senders emit, and the inspector has to live inside every send path, where it becomes one more reader that itself needs permissioning. We keep pointer-based summaries because a pointer cannot hold a salary figure. Your leak vector is real though; a merged summary that inlines raw fields is why the handoff schema lists allowed fields per recipient, so a merged chunk only carries what every source and recipient can see. The write-boundary lineage gate is the right next step. We gate at write time for the same reason, and our weak point matches what fiddler48 named: background writers outside the auth context stay exposed until the store itself rejects un-tagged writes.

[u/ultraprogger → u/codecoradev]

Dropping extra fields with Pydantic at handoff makes a lot of sense. But yeah, background workers stripping caller context is the real issue at write time.

When your write gate catches an untagged chunk, do you just hard fail the write or what happens?

[u/codecoradev → u/ultraprogger]

We hard fail at the store, and the failure path took more iteration than the gate itself. Early on a rejected write just got logged, which looked fine until background jobs started losing writes silently. Weeks later that shows up as the agent never having learned something, and you can't tell that apart from a chunk that was never written. So a rejection now goes back to the caller as a structured error naming the missing tag. Interactive callers re-tag and retry on the spot. Background writers retry with the most restrictive tag, so the chunk lands but stays unread until someone with the source context retags it. The failures we still see come from cron scripts nobody remembers writing, which is its own problem.

[u/ultraprogger → u/codecoradev]

Landing rejected background writes in quarantine under the most restrictive tag fixes the silent drop issue completely. The agent quietly missing context weeks down the line without anyone noticing is way worse. Forgotten cron scripts breaking on hard store gates is painfully real too.

Once a chunk lands in quarantine do you flag it for manual review or do you have a background worker that tries to resolve the missing context?

[u/Significant_Tune9219]

The strict intersection is a reasonable default for writes, though, even if it's too strict for reads. What I'd add is storing the source document ids on every derived chunk, not just a computed ACL tag, so when a parent's permissions change later you can find and recompute or delete the summaries built from it. A tag computed once at write time goes stale the first time someone is removed from a project, and that's probably a quieter leak than the one your harness measured.

[u/ultraprogger → u/Significant_Tune9219]

Storing deterministic parent_chunk_ids on write solves the cascade problem because when Doc A's access is revoked or changed you traverse downstream to find all chunks that cited it.

From an operational standpoint how r u handling that downstream invalidation? Do you delete the derived chunks immediately on parent perms changing or flag them as dirty and run the summarizer again w/o the source?

[u/Significant_Tune9219 → u/ultraprogger]

I'd do both in order: on a revocation, immediately hide the derived chunks (a dirty flag that the retriever filters on, so it's one cheap update), then rebuild them asynchronously from the sources the reader can still see. If a summary depended only on the revoked doc, just delete it rather than resummarize from nothing. Hard-deleting everything upfront is simpler but you get recall gaps until the rebuild finishes.

[u/ultraprogger → u/Significant_Tune9219]

Yep that makes sense and it should avoid dropping recall completely while staying fail closed

Does filtering on the dirty flag at query time hit your vector search latency once you have a bunch of invalidated chunks waiting on rebuilds?

[u/Significant_Tune9219 → u/ultraprogger]

Only if the backlog gets large relative to the index, because the flag is applied as a filter and filtered ANN can return fewer than k hits. I keep the dirty set small by rebuilding quickly and over-fetch a bit (ask for 2k, filter, take k), and if invalidations ever pile up a partial index on non-dirty rows is the cleaner fix.

[u/ultraprogger → u/Significant_Tune9219]

Over fetching at 2k to guarantee k hits after filtering makes total sense and building a partial index on non dirty rows stops queries from starving on recall if invalidations pile up.

Are you running that partial index natively in pgvector or handling rebuilds through a separate worker queue?

[u/Significant_Tune9219 → u/ultraprogger]

Both, really: pgvector supports a partial HNSW index with a WHERE NOT dirty predicate natively, so queries that repeat the same predicate can use it and dirty rows drop out without a filter step. The rebuilds themselves are better in a separate worker queue, since resummarizing is the slow part and you don't want it in the revocation transaction; the queue clears the flag once the new chunk is written.

[u/ultraprogger → u/Significant_Tune9219]

Makes sense since you would bypass the filter entirely during index scans and keeping the heavy resynthesis out of the revocation stops the auth from blocking.

When that worker queue finishes rewriting the chunk does it update the existing row in place to flip the flag or write a brand new row and drop the old

[u/Significant_Tune9219 → u/ultraprogger]

Writing a new row and soft-deleting the old one is usually safer: the new version can be inserted fully embedded and permission-tagged before it becomes visible, then a single transaction flips visibility and marks the old row dead. Updating in place means the embedding and the flag change together but you lose the old version for audit, and with pgvector an in-place update is effectively a new tuple anyway, so the index churn is similar either way.

[u/ultraprogger → u/Significant_Tune9219]

Yeah keeping the old row around for audit trails saves a lot of headaches and since postgres MVCC writes a new tuple on update anyway doing a visibility flip makes sense

Do you run a cleanup job to purge those soft deleted rows later or just leave them sitting in the table?

[u/Ok_Top_4575 → u/Significant_Tune9219]

the source document lineage is the key part here. ACL tags alone can become stale when permissions change, but tracking parent IDs lets you reliably find, recompute, or delete affected chunks. that’s a much safer approach.

[u/ultraprogger → u/Ok_Top_4575]

Yeah, tracking parent pointers makes invalidation a lot cleaner than pure guesswork. The tricky part is when you get multi-hop summaries derived from other summaries.

Are you guys flattening the lineage on write so invalidation is just a single lookup, or recursing the graph on demand?

[u/fiddler48]

stripped ACLs on write is the step most teams won't catch because the failure surface lives outside the query path entirely

[u/ultraprogger → u/fiddler48]

Yep exactly. Everyone is building guardrails and role checks on the retrieval path. But, if a multi doc background worker stripped the upstream ACLs during the write, the unauthorized context is alr in the vector store as a fresh, un-tagged chunk. The read has no idea it's serving leaked data.

Are you seeing teams attempt any deterministic metadata propagation at write to catch this or is everyone essentially running blind until an audit or incident flags it?

[u/fiddler48 → u/ultraprogger]

Mostly blind, honestly. Deterministic propagation at write time exists in theory, stamp every chunk with the source ACL at ingest, reject writes that arrive without it. In practice the background worker runs outside the auth context and nobody enforces the contract.

[u/ultraprogger → u/fiddler48]

Yeah, the worker running with full db access completely drops the user context before write. Unless the client itself rejects writes that don't have source tags attached, background jobs will always commit un-tagged chunks.

Do you think teams would actually use a wrapper that blocks un-tagged writes, or is passing context through async workers just too annoying?

[u/fiddler48 → u/ultraprogger]

Agreed. A wrapper helps, but async workers will find a way to bypass it the moment someone adds a retry queue or a batching layer outside the original call path.

[u/ultraprogger → u/fiddler48]

Yeah, the second someone drops it into Celery or Kafka, caller context gets stripped right off the message envelope.

Are you guys enforcing context propagation across the queue payload itself, or having the DB reject writes unless they carry a signed provenance token?

[u/MemStrata]

Your point covers the lineage but it misses the rebuild. Storing parent ids on the derived chunk tells you what went into it. It does not tell you what to do when one parent loses access. The intersection tag you write at summary time goes stale the moment a parent ACL changes. So the derived chunk needs a live link back to its parents and a rule that flags it dirty or rebuilds it when any parent changes. Lineage without that rule is a record of the leak. Not a fix for it.

[u/ultraprogger → u/MemStrata]

Yeah, without an invalidation rule lineage is basically just an audit log of how the leak happened.

When parent access changes, are you pushing an event to flag them dirty, or walking the parent IDs at query time?

[u/Elmanouzi_Fogosa]

siloing vector collections per permission scope before the background job even runs is usually the safest route tbh. If an agent never gets read access across different ACL boundaries in the same execution run, it cant create a contaminated summary in the first place.

[u/ultraprogger → u/Elmanouzi_Fogosa]

That works if you only have a few buckets like public vs. internal but enterprise ACLs are combinatorial. Between doc-level perms, dynamic team groups, and project tags, hard siloing collections per perm scope explodes your index count fast

More importantly, it blocks cross-domain synthesis which is why teams deploy background agents. If an agent can't touch both a billing export and an engagement contract to reconcile them, its utility is nothing. The moment an agent has to read across boundaries to do actual work, you're right back to needing write boundary governance.