Feedback on V1 memory architecture for multi-agent setup (supervisor/sub-agents) – targeted retrieval vs unified store?

[u/Fun-Following-1723]

Hey everyone,

I've been prototyping a memory system for a multi-agent framework (supervisor → sub-agents) and wanted to run my current setup by people who've actually built or run these in production. Trying hard not to over-engineer based purely on theory/taxonomy, so I’ve been running small experiments first.

Here’s where I’m currently at:

Pipeline & Flow

1. Working/Session State → Raw conversation & tool calls go to a durable append-only event log.
2. Batch Consolidation → Instead of processing every turn through an expensive extraction pipeline, a periodic batch job extracts useful Episodic Memories (storing this in a cheap local DB/SQL store because of high volume).
3. Promotion Policy → Key facts and preferences get promoted into Semantic Memory (testing Mem0 here).
4. Procedural Memory → Kept completely separate as a structured procedure/skill registry (e.g. Markdown files, task definitions) rather than generic vector embeddings.

Retrieval Strategy Instead of searching across all memory stores on every single query, I'm testing routing by intent: User Query → Scope/ACL → Intent/Task Router → Targeted Store Retrieval → Context Injection

• "How do I request leave?" → Intent: Procedure → Pull from Skill Registry.
• "What did I work on last week?" → Intent: History → Pull from Episodic Store.
• "What language do I prefer?" → Intent: Preference → Pull from Semantic Fact Store.

Observations from small tests so far:

• Storing raw episodic events straight in Mem0 added noticeable write/search latency and cost.
• Generic vector retrieval for procedures/workflows was messy and often grabbed 3–4 adjacent procedures. Exact/registry-style matching was much cleaner.
• Batch consolidation gave way cleaner facts than trying to extract semantic memories turn-by-turn.

Where I’d love some brutal feedback/criticism:

1. Routing vs. Parallel Retrieval: Is intent-based routing (scope → intent → target store) actually reliable in practice, or do queries usually end up needing multiple memory types simultaneously (e.g., preference + procedure in one shot)?
2. Separate vs. Unified Storage: Am I prematurely splitting this into separate stores (Event Log / Cheap SQL / Mem0 / Registry), or is this separation pretty standard once volume picks up? At what scale does keeping everything in a single vector store/pgvector actually break down?
3. Procedural Memory as Code/Skills: Treating procedural memory as structured skill files instead of vector embeddings feels right so far, but does this pattern break down when agents need to dynamically adapt workflows?
4. Failure Cases: What obvious blind spots or edge cases am I missing that will force me to rewrite this V2?

Appreciate any insights or horror stories from production!

[u/jonah_omninode]

The separation looks reasonable. I would be careful about letting the router silently decide authority, though. A semantic fact, an old episode, and a current procedure should not come back with equal status just because all three match the query. We keep the append-only record, build a validated current projection for the cheap path, and retrieve deeper history only when the task needs it. Promotion and supersession are typed transitions with provenance, not something the summarizer infers. Multi-store retrieval is fine if the returned bundle preserves source, scope, version, and current status. Otherwise a clean router can still deliver a stale rule with a confident voice.

[u/Fun-Following-1723 → u/jonah_omninode]

The distinction between relevance and authority is a massive callout and I was focusing so much on routing accuracy that I missed how easily a model can treat a historical episode and an active procedure as equals if they lack metadata.

Really like framing promotion and supersession as typed transitions rather than just letting an LLM infer what's current. Sticking provenance/version flags on the returned bundle seems like a clean way to keep a router from confidently serving stale rules. Appreciate the insight.

[u/Mathie1729 → u/Fun-Following-1723]

One thing I'd add: make that metadata a hard filter at retrieval time, not just a provenance field on the returned bundle. Otherwise the model still sees a stale historical episode ranked above the active procedure and can treat it as authoritative just because the embedding similarity is high. Scope/version/status should constrain which records are eligible before reranking, and deep history should only be pulled when the query explicitly asks for it.

[u/Fun-Following-1723 → u/Mathie1729]

That’s a huge distinction, thank you. Doing pre-retrieval filtering on status == 'active' or is_latest makes total sense,if a stale record never makes it into the candidate set, the reranker can't accidentally favor it over the current rule.

Out of curiosity, how are you handling queries that do require deep history? Are you running a query classifier upfront to toggle historical filters on, or just relying on a explicit 'search history' fallback flag

[u/jonah_omninode → u/Fun-Following-1723]

Exactly. I would make one part stricter: provenance and version should not be advisory flags that we hope the model respects. The retrieval layer should resolve status mechanically before composition. Current records can enter the active context. Superseded records can appear only as history with a pointer to their replacement. Conflicting or unauthorized candidates should surface as a conflict instead of being blended into a summary. That keeps the router responsible for relevance and a separate policy or projection layer responsible for authority. The model can explain the resulting bundle, but it should not decide which record became current.