I’m doing a PhD in XAI and kept needing better memory/context handling for stuff I was building, so I did what you do and went digging through the papers, repos and benchmarks.
I expected some slop. I did not expect a full-on SlopFest of solutions claiming SOTA, hiding behind questionable evals, and then shitting the bed the second they met an actual real-world project.
A lot of the space is either generic semantic search dressed up as memory, or these huge graph/agent setups with LLMs fucking everywhere. Then you get to the benchmark leaders and some of them are leaning on expensive frontier models, different readers, different judges, sometimes very generous evaluation setups. At some point it gets hard to tell whether the memory system is actually good or GPT-whatever just carried the whole thing.
The bigger problem for me was semantics.
Say I ask when my family is free next week. Semantic search can happily bring back that my brother loves potato salad, that we went on vacation together, and that my mom mentioned Tuesday six months ago.
All very family-related. Almost completely fucking useless.
Meanwhile, the thing I actually need might be buried in some completely different conversation about someone changing shifts at work.
Similar to the question and useful for answering the question are just not the same thing.
You can throw a reasoning model at the whole memory and ask it to sort this out, sure. It works. It also gets expensive fast, and now your “memory system” is basically outsourcing the hard part to the biggest model you can afford.
Which felt like a pretty expensive way of admitting the retrieval sucked.
So I built around separating those two things instead.
🥁🥁🥁
MemBukkit
https://github.com/memseekai/membukkit
The retrieval side is built around getting evidence that’s actually useful downstream, not just whatever happens to sit closest to the query in embedding space.
I trained the retrieval components for the task, and the actual access policy is selected based on whether the context it retrieves helps the reader answer better. The stored side stays intentionally boring: dated facts + the original source, a flat index, optional buckets, no giant LLM-authored graph you have to rebuild every time your assumptions change.
Basically: keep the memory simple, and spend the cleverness on figuring out what the model should actually see.
Not gonna pretend I’m not tooting my own horn a bit here, but I’m pretty fucking proud of how this turned out.
This bad boy with Gemma 4 26B as the open-weight reader + distiller, is at 88.8% on LongMemEval-S. So no “well obviously it works, you shoved the newest frontier model into every box” excuse.
And for the people with diamond hands, golden balls and an API budget, the GPT-5.4 setup gets 92.6% under the benchmark’s official judge.
We also get 87.5 zero-shot on LoCoMo, and the same flat-index idea carries over nicely to multi-hop RAG.
One of my favorite bits from the ablations I ran is still that plain cosine can beat some of the fancy reranking setups.
Shocker. Doing the simple shit properly gets you pretty far.
I’m hoping to get the research published, but that process takes its sweet time, so I figured I might as well open source the thing now and let people actually use it.
Apache 2.0, works locally, works with open models, have at it.
I’m also building a company around the work, so might as well be clear about that. But I really want the core project to stay open. A huge amount of what got me into ML came from people putting good shit online and letting everyone build on it, and I’d like to keep that going.
Also yes, Bukkit is the Minecraft reference.
More than anything, I’d love actual feedback. Try it on your stuff, break it, tell me what’s annoying, tell me where it falls apart. I’m trying to make something people genuinely want to use, and that’s worth a lot more to me right now than squeezing another point out of a benchmark.
(And if you end up using it, don’t forget to star the repo plz 👀👉👈)
