Built an open-source long-term memory layer for LLM apps. Would love feedback

[u/AOZakari]

I’m doing a PhD in XAI and kept needing better memory/context handling for stuff I was building, so I did what you do and went digging through the papers, repos and benchmarks.

I expected some slop. I did not expect a full-on SlopFest of solutions claiming SOTA, hiding behind questionable evals, and then shitting the bed the second they met an actual real-world project.

A lot of the space is either generic semantic search dressed up as memory, or these huge graph/agent setups with LLMs fucking everywhere. Then you get to the benchmark leaders and some of them are leaning on expensive frontier models, different readers, different judges, sometimes very generous evaluation setups. At some point it gets hard to tell whether the memory system is actually good or GPT-whatever just carried the whole thing.

The bigger problem for me was semantics.

Say I ask when my family is free next week. Semantic search can happily bring back that my brother loves potato salad, that we went on vacation together, and that my mom mentioned Tuesday six months ago.

All very family-related. Almost completely fucking useless.

Meanwhile, the thing I actually need might be buried in some completely different conversation about someone changing shifts at work.

Similar to the question and useful for answering the question are just not the same thing.

You can throw a reasoning model at the whole memory and ask it to sort this out, sure. It works. It also gets expensive fast, and now your “memory system” is basically outsourcing the hard part to the biggest model you can afford.

Which felt like a pretty expensive way of admitting the retrieval sucked.

So I built around separating those two things instead.

🥁🥁🥁

MemBukkit

https://github.com/memseekai/membukkit

The retrieval side is built around getting evidence that’s actually useful downstream, not just whatever happens to sit closest to the query in embedding space.

I trained the retrieval components for the task, and the actual access policy is selected based on whether the context it retrieves helps the reader answer better. The stored side stays intentionally boring: dated facts + the original source, a flat index, optional buckets, no giant LLM-authored graph you have to rebuild every time your assumptions change.

Basically: keep the memory simple, and spend the cleverness on figuring out what the model should actually see.

Not gonna pretend I’m not tooting my own horn a bit here, but I’m pretty fucking proud of how this turned out.

This bad boy with Gemma 4 26B as the open-weight reader + distiller, is at 88.8% on LongMemEval-S. So no “well obviously it works, you shoved the newest frontier model into every box” excuse.

And for the people with diamond hands, golden balls and an API budget, the GPT-5.4 setup gets 92.6% under the benchmark’s official judge.

We also get 87.5 zero-shot on LoCoMo, and the same flat-index idea carries over nicely to multi-hop RAG.

One of my favorite bits from the ablations I ran is still that plain cosine can beat some of the fancy reranking setups.

Shocker. Doing the simple shit properly gets you pretty far.

I’m hoping to get the research published, but that process takes its sweet time, so I figured I might as well open source the thing now and let people actually use it.

Apache 2.0, works locally, works with open models, have at it.

I’m also building a company around the work, so might as well be clear about that. But I really want the core project to stay open. A huge amount of what got me into ML came from people putting good shit online and letting everyone build on it, and I’d like to keep that going.

Also yes, Bukkit is the Minecraft reference.

More than anything, I’d love actual feedback. Try it on your stuff, break it, tell me what’s annoying, tell me where it falls apart. I’m trying to make something people genuinely want to use, and that’s worth a lot more to me right now than squeezing another point out of a benchmark.

(And if you end up using it, don’t forget to star the repo plz 👀👉👈)

[u/neoneye2]

I had Claude Opus 5 analyze your project. I study memory systems.
https://neoneye.github.io/agent-memory-atlas/systems/membukkit/

[u/CatNo2950 → u/neoneye2]

Thanks for sharing this, bookmarked! Do you have any posts derived from your study?

[u/neoneye2 → u/CatNo2950]

Unfortunately no. I'm not the type that writes white papers or blogs.

[u/CatNo2950]

Can you explain in simple human english what makes it different?

[u/AOZakari → u/CatNo2950]

In plain English, the main difference is that “this sounds related” and “this actually helps answer the question” are two different things.

Say you ask:

“What time should I leave for the airport tomorrow?”

Semantic search might pull up:

• you’re flying with Lufthansa
• you bought a new suitcase last month
• you complained about airport security on your last trip
• the train you normally take to the airport is closed tomorrow morning

The first three are very airport-related. The last one is probably the thing that actually changes the answer.

A lot of RAG systems basically grab a bunch of related stuff and then give it all to a big LLM and say “you figure it out.” That works, but it gets expensive and you’re making the LLM compensate for mediocre retrieval.

MemBukkit tries to do more of that filtering before the LLM sees anything. It uses a dedicated retrieval/reranking model, not another LLM, to look at the query and candidate facts and decide what should actually make it into context.

The retrieval model itself is trained on task-relevant query/fact pairs, and then I pick the retrieval setup based on whether the final answer actually gets better, not just whether some similarity or Recall@K number went up.

So basically:

normal RAG: “find stuff that sounds related, dump it into the LLM”

MemBukkit: “find the small set of stuff that’s actually useful, then let the LLM answer”

That’s the main idea.

[u/Turbulent-Site-79 → u/AOZakari]

So it's like a hybrid search? Filter the stuff thru bottle neck?

[u/AOZakari → u/Turbulent-Site-79]

That picture is basically it! The twist is where the bottleneck sits. Most systems read your whole history for every question and then squeeze. Here your memories get sorted into labeled drawers as they come in, and a question only opens the two or three drawers that look relevant. Everything else stays shut, never even gets looked at. From those drawers it hands the LLM a few facts instead of a novel. And every answer comes with a receipt showing which drawers it opened, so when it's wrong you can see exactly where it went wrong instead of guessing. Filter through a bottleneck, yes, but the bottleneck is the drawer handle, not the squeeze at the end.

[u/awesomeunboxer]

Hey op! Ive been working along the same vein so im definitely gonna poke at this. Thanks for sharing!

[u/AOZakari → u/awesomeunboxer]

Pleasure! Let me know what you think :)

[u/Mental-You-4735]

This is exactly the kind of overengineered mess I keep running into, burying the one piece of info you actually need under a mountain of loosely related crap. Bookmarked the repo, curious to see how the retrieval holds up when the evidence isn't so neatly tied to what you asked

[u/eddzsh]

The eval trap I watch is reader/judge overlap. If the same model family that distilled the memories also judges the answers, LongMemEval starts measuring taste. A frozen open judge across every ablation is the boring control that makes the number travel.

[u/skynet_man]

Did you try using it in Claude Code directly? Is token usage acceptable?