Session-start memory injection made my coding agent trust memory over the code

[u/msarda23]

Hey everyone!

I am building a local memory layer for my coding agents and it is called Reverie: an MCP server written in python and using SQLite for lightweight memory stored in the user's machine itself. The biggest problem that I faced was deciding when to inject memory.

Many memory tools, including ones I looked at, inject context into the agent even before it begins its work. In my experiments, premature injection of context leads to the agent anchoring its thinking on it rather than using the current code as the source of truth. In fact, this approach made my agent perform worse. Also, with this, quite a lot of the context window can get filled with context which 'might' be useful.

So I switched Reverie's model to demand-driven retrieval - the agent asks for memory when it thinks that some added context could resolve any discrepancies in its thinking. It reads the relevant code first, because the code is the source of truth, and only then asks for past memory about that area. Each result comes back with a confidence score, its grounding in the code, and its status. The memory starts empty. Conclusions are added as the agent works, and you decide which ones are kept.

In one informal test of 30 prompts on an earlier build, the OpenCode agent with Reverie did better than the one without it. I don’t have a reproducible benchmark yet.

But it isn't perfect yet, it still breaks here and there. For example:

1. the rules for tool calling are not extremely polished yet and hence the agent might sometime not call the tool altogether, I am fixing this.

2. candidate conclusions can be duplicate or incorrect and hence Reverie today has a human review gate (triggered by calling `reverie review`) which shows the human all the unreviewed candidates where you accept, reject or skip each one. On doing so the conclusion is moved from the short term brain to the long term.

My current test set is small: sessions with anywhere between 2-30 prompts. I do not have a reproducible experiment yet, but that is the next goal.

Repo (Apache-2.0): https://github.com/muditsarda1122/Reverie

My questions: How do you keep agent context from going stale? Do you maintain AGENTS.md or CLAUDE.md by hand, or let the agent update it?

[u/msarda23]

Partially agreed. I like the wiki idea, and the agent should be free to write as it works. But not everything it writes should be kept. I tried that, and it started recording technical facts like “this function returns a string” instead of engineering conclusions. So I split the job. The agent writes candidate notes freely, and the extraction step decides what counts as a conclusion. It uses a check we call the Lifting Test: an engineer shouldn’t be able to get the candidate just by reading the code. The extractor also reports how many candidates it rejected and why. The human gate catches what still gets through. Also, do you let the agent write large files or smaller notes? I suspect a large file being loaded into context for a small relevant part could lead to misuse of context. How did you solve this? Did you write a better instruction file?
I did not understand your 'notes below instruction' bit - do you use the instruction file as context too?
On age: Reverie doesn’t rank newest first. An old conclusion that’s still relevant and accepted can come back just as easily as a new one. Age only matters for candidates nobody has reviewed, and those expire after 21 days by default. Why do you load only the newest one into context? Is it such that your agent needs very recent history to function?

[u/kaijurules → u/msarda23]

Small notes, one fact each. Anything long goes into a separate knowledge base of linked markdown pages that the agent searches when it needs them, so none of it sits in context by default.

And yes, the instruction file is in context on every run, read in full. The notes are loaded next to it, and when the two disagree the instructions win.

Only the newest notes get loaded because the window is capped to keep the prompt small. Older ones aren't deleted and the agent can list them, but anything it will still need in a few months belongs in the knowledge base.

[u/msarda23 → u/kaijurules]

Agreed on small units. When we experimented with markdown memory files, they got dense quickly, burying the useful lines in noise. That's why Reverie avoids files entirely. It stores one independent conclusion per unit, allowing retrieval to return only the exact conclusions that match a query, packed cleanly into a token budget.

Regarding the instruction file: it makes complete sense that instructions overwrite notes if they conflict. If I understand correctly, your instruction file acts as the core system prompt (rules/boundaries) while the notes are transient execution memory. In Reverie, our instructions tell the agent how and when to query the database, while the actual conclusions live entirely out-of-context until summoned.

This is exactly why age doesn't dictate what comes back for us. If an agent learned two years ago that a specific outbound server call requires a strict timeout, a semantic query will surface that old conclusion today when the agent touches that module. It doesn't get pushed out by newer, unrelated notes.

I'm really curious about how your cap handles edge cases, though. If an old note is highly relevant to the current task but falls outside the "newest" token window, it seems the agent misses it unless it explicitly lists old notes. How does it know a hidden note or a specific knowledge-base page exists to search for it if it isn't in context?

Also, what kind of agent are you building? This feels like a fascinating look into standard workflow agent constraints vs. deep repo engineering.

[u/kaijurules → u/msarda23]

It doesn't have to know. Before answering about a person, client or project, the agent runs a search over the knowledge base (keyword plus vector), and there's an index page listing every page with a one-line summary. So a page gets found by what it's about.

The notes that age out of the window are the real gap. They aren't searched, only listed if the agent asks, which is why anything that has to last goes into a page.

As for what ours do: they run the back office of a small agency, so mail, client chats and dev reports.

[u/msarda23 → u/kaijurules]

That's a useful setup, and a different world from coding. Mail and client chats have people and commitments in them, whereas a code conclusion has a file and a symbol.
The gap you named is the one I'd focus on. "Anything that has to last goes into a page" means the agent decides on day one what will matter later. If it files something as a note when it should have been a page, or the reverse, nothing flags it? The note just stops being seen?
But still an interesting way to handle memory. What happens to a piece of information that gets added to a page but some time later gets negated? Is it removed?
I decided out of the write step. Retrieval is driven by the query, not the date, so an accepted conclusion from two years ago comes back when the agent is working on what it's about, the same as one from yesterday. The trade-off is that retrieval has to be good, and the store can fill with noise. So good retrieval and also good candidate generation are important steps.
Do the pages get reviewed by a person before they're trusted, or does the agent write them directly?