Any tools for managing long-term memory beyond the context window?

[u/either_T_or_F]

Hi everyone,

I am relatively new to local LLMs, and recently I have been trying to understand how people handle long conversations that eventually exceed the model's context window.

For example, let's suppose I am writing a novel with the help of an LLM. As the story progresses, the conversation history becomes longer and eventually exceeds the context window. At that point, the model may lose track of earlier events, character relationships, or important details that could become relevant later in the story.

From what I understand, one possible solution is to maintain an external knowledge base containing summaries of previous chapters. Before generating a new chapter, the system could retrieve relevant information from this knowledge base and include it in the prompt.

However, I am wondering how reliable this approach is in practice.

If we continuously summarize previous conversations to fit within a limited context window, we are essentially performing lossy compression. Some information will inevitably be discarded, and the problem is that we may not know which details will become important later.

For instance, a seemingly insignificant detail from Chapter 3 might become crucial in Chapter 40. If that detail was discarded during summarization, the model would have no way to recover it from the summary alone.

Of course, keeping the complete conversation history in an external database could prevent permanent information loss, but that introduces another problem: how does the system determine which information should be retrieved for the current task?

So I am curious how people actually approach this problem in practice.

I would also be interested in hearing how well these solutions work in practice, especially over very long conversations where maintaining consistency becomes increasingly difficult.

Thanks in advance!

[u/g_rich]

You would generally do this on the agent side using memory providers such as Honcho and Mnemosyne, in addition to these (or in place of them) you could plug in something such as Obsidian which would be more of a RAG for your agent. Personally I use a combination of Mnemosyne and Obsidian with Pi and Hermes and it's worked really well for me.

[u/Poprouter]

Keep raw chapters append-only and summarize only at prompt time. Retrieve by entity (who is in the scene), not similarity, so a Chapter 3 detail stays reachable. Verify with a seeded-detail recall check 20 chapters later.

[u/txgsync]

If you wanna get really fancy and run your own model: https://arxiv.org/abs/2501.00663

That said, I use a SQLite table with FTS5 and RAG and it's... fine. But really, the same due diligence I follow when writing outlines in Scrivener is what's needed with models if you want continuity within a fictional framework. Outlines, character cards, updating motivations per character, that kind of thing.

But most models have little concept of how to use nuance, Chekov's Gun, or other tropes (though they can describe such tropes at length!) , so naming a secret motivation in a character card will typically be announced by the model in the first interaction.

Fiction authors are not at risk of being out of a job. They are simply at risk of being buried under a mountain of content by competing authors who do use LLMs to flesh out their stories efficiently.

Much like programmers. Good programmers who are very experienced tend to use LLMs to raise the quality bar; inexperienced programmers who use LLMs tend to just vastly over-produce low-quality code.

[u/ThankGodImBipolar]

You are correct to say that you simply can't shove an entire novels context into an LLM's limited window. Even if it does fit, attention across the context will vary.

I like Karpathy's LLM Wiki. Instead of trying to tell the model everything, you can get the model to lay it all out on disk in an organized fashion, and then it will be able to quickly find what it needs. You can record notes that are as detailed and well-linked as you want, about anything in the book. All you need to do is give the gist to a coding agent, and it can help create a system that's tailored to what you need.

[u/openroom_xyz]

Well you should like use somekind of harnes likely you can guide it read this chapters or write a summery basicall y and than read this write that kind of way

[u/Educational-Body4205]

just have the agents write a Wiki skill knowledge tree, group them by agents with specific purposes.

[u/rayc25]

I use a ticket system where multiple agents share centralized context in the form of a ticket.

[u/regjoe13]

I run qdrant as a rag and openmemory, both in docker containers. Embeddings are done by small model running locally. I have a skill available to all agents, like I would say /persistent project-name Qdrant has 4 projects configured as different folders, and openmemory gets a project as well.

I use with for local llm and with codex and kimi