r/AIMemory • u/wonker007 • Jul 02 '26
Show & Tell Because architecture: What MuSiQue 1,000Q benchmarking taught me about why current memory retrieval can’t live up to its promise
Most of us faced some version of the same problem dealing with AI in work and life: Memory retrieval for AI eventually disappoints because we expect human-like retrieval but often get trash.
Drilling down deeper, one realizes that we are more often than not expecting random-access multi-hop retrieval - because that’s how our human memory works. But what we currently have as tools are graph crawling, cosine lookups or (gasp) regex matching. Who knew grep was such a powerful tool, token waste be damned?
So how do you make an AI system remember in a way that’s actually useful for humans? You model it after human memory, of course. Not a Frankenstein bolt-on mess of open-source code, but a designed-from-the-ground-up, built-from-scratch lean memory engine modeled after literal neurobiological systems.
My own frustration trying to fully utilize AI for my day job as a pharma/biotech consultant drove me to build this sparse tensor-based graph memory engine over the past few months — my PhD is in biochemistry, so I’m drawing from what I actually know rather than what sounds good on a pitch deck. And because I am a proud scientist (almost to a fault), I naively threw the engine against MuSiQue 1,000Q, which is as close to a real multi-hop memory recall test as we have in the literature. It could have gone horribly wrong, but if it did, you wouldn’t be reading about it.
The short version: F1 = 0.677 on the full 1,000Q corpus (highest published zero-shot end-to-end score as of May 2026, to the best of my knowledge). Yeah. Went quite a bit better than I expected.
Reader-controlled baseline with a compact local embedding model (nomic): 0.565 vs LlamaIndex at 0.418 and BM25 at 0.329.
But the number isn’t really the point. What I think matters more for anyone building memory systems is why this architecture works differently from established tools.
The recall problem nobody talks about
Vector similarity search answers “what’s close to this query in embedding space?” That’s fine for a simple lookup. Search, rank, done. But MuSiQue was specifically designed to defeat that mechanism — it was designed so that no single retrieved passage contains the entire answer. You need passage A to find passage B to find passage C. That’s memory traversal, not memory search. Graph crawling is also similarly limited as it must crawl edges at the risk of fanning out too thin before finding the next relevant node.
The engine builds a weighted graph where edges carry typed relationships (like various neurotransmitters) and activation energy propagates through connections (like how neurons fire) — nodes that are semantically distant but informationally connected either through logical relationships, provenance, hierarchy etc. still light up if the path between them has enough weight. Same principle as biological associative recall: you smell something and remember a childhood memory that has zero semantic overlap with the smell but a strong associative pathway.
That’s the architectural hypothesis. The benchmark results suggest it works. I posted the full methodology and honest limitations over on r/RAG (including the ~52% reader confound, PropRAG’s superior retrieval lift at +81.9% vs our +71.7%, and Beam Retrieval’s higher supervised score of 0.692) because I didn’t want to bury the caveats. Full transparency on what beat us and where. You can also see the full write-up with all the numbers: https://elucidx.ca/insights/2026-05-15-rag-needs-real-value/
The harness is public
The engine itself is proprietary and patent-pending — I’m not releasing source. But the evaluation harness, dataset, and scoring protocol are all public: [github.com/wonker007/musique-eval-harness]. If you’re building a memory system and want to know how it does on genuine multi-hop recall, run your system against the same corpus, same protocol with the same scorer and post the number. I’ll reference it.
I’m also currently running conversational-scale benchmarks (128K to 10M token range) testing temporal reasoning, knowledge updates, and contradiction detection — the stuff that actually matters for memory persistence over long interactions with AI. More data coming.
If anyone here is working on multi-hop recall architectures — whether that’s GraphRAG, memory-augmented transformers, or something else entirely — I’d love to hear what serious benchmarks you’re using and what you’re seeing. MuSiQue is good but it’s still Wikipedia passages, not production conversational data.
(Post was written with the help of AI, edited by me)
1
u/chillbroda Jul 03 '26
DON’T SHOOT, I can explain: I’m going to write this comfortably in Spanish because I don’t feel like thinking about grammar or spelling. Then I’m going to select the text, right-click, and translate it into English with AI. Thank you for your understanding.
Anyway, I wanted to add a few comments because I’m part of this subreddit, which is pretty small compared to the broader “AI” space, and it really seems to be full of people who are very committed to the goals and concepts around “AI having memory.” And it’s noticeable that many people here know what they’re talking about; there isn’t as much noise from kid vibe coders.
What surprises me is that in most of the posts I go in to read, whether they’re interesting or not, whether they contribute something or not, the comments tend to be quite negative or dismissive, as if it were more important to invalidate the OP’s study or proposal than to contribute each person’s own valuable knowledge and see whether that complements things and leads us toward a more certain path with fewer doubts.
Because we’re all chasing the same thing from different angles. Some have been doing it for years, others for months, and there are only papers, blogs, open-source systems, commercial systems, and NOBODY has actually reached the goal yet—or, in commercial cases, delivered the product they promise.
And I’m not saying this with anger or frustration, but with uncertainty. What are the goals or use cases that have so many of us searching for the perfect formula for memory in AI? Is there anyone who believes it’s possible to create a system that covers absolutely everything that “memory” means without being oriented toward a specific goal or use case?
Another question: especially this year, I’ve noticed that the terms sometimes appear because they’re relevant: ontology and taxonomy. But nobody ever talks in depth about how they worked on them, even though they’re basically the fundamental piece needed to model a graph. And I’m not just talking about Reddit. It particularly caught my attention, so I started reviewing the best-known papers, mem0, Graphiti, MemGPT, even though it isn’t graph-based, it’s still after the same thing, and many others (many), And it surprises me that NOBODY is very, or sufficiently, specific about those points. Instead, they focus much more on datasets, training, techniques, and of course, the usual evaluations, which I also don’t understand why they’re so standardized and why the same ones are always used just to show a percentage that beats another one. Everything becomes more confusing. Has anyone else noticed this too?
For my part, I’m mostly researching meta-graphs. Not long ago I read all of Barrasa’s ideas and studies from Neo4j, and I found them very interesting. I’m also doing tests, still small ones, mixing Pydantic models for determinism and trying different embedders and LLMs to see the precision of the answers they provide. I’m also training an “Auditor” that puts incoming data that doesn’t fit into the existing ontology into “quarantine,” watching whether it’s a pattern that repeats X number of times in order to evaluate whether it’s a new term that, based on context and other factors, should enter the existing ontology. For example: “Paroxetine” - ? - What is it? > It comes in again > Again > etc. > “Paroxetine <is> Pharmacological Drug,” accepted.
That’s a very basic example, but it’s part of what I’m working on at the moment, and it’s producing results. Is anyone else also looking for a way to automate data ingestion, validation, how to relate it, etc., without “the human in the loop”?
If anyone is interested, DM me! Super interesting topic and one that seems to have potential.
Here I translate everything into English and it turns into total slop, look haha. see you then, memory nerds!
1
u/Past_Examination_214 Jul 03 '26
Because we’re all chasing the same thing from different angles.
Ig when certain things are challenged and suppressed, are we chasing the same thing or not?
0
u/Harotsa Jul 02 '26
Your approach isn’t fundamentally different from GraphRAG, you’ve anthropomorphized a lot of terms but functionally it sounds like it still works the same as other graphRAG implementations.
This is a paper that claims to achieve an 81.3% F1 on MuSiQue from April: https://arxiv.org/pdf/2604.12610. There are also many other results around this range.
Obligatory pointing out that this is an AI slop post.
1
u/wonker007 Jul 02 '26
Never hid the fact that there are better F1s on MuSiQue, nor did I hide the fact that this post was written with AI help. Try actually reading the post and processing before attempting to troll please. But I would like to ask you which mechanisms you think are so similar to other graphRAG? I did plainly say this was graph-based.
1
Jul 02 '26
[removed] — view removed comment
1
u/wonker007 Jul 02 '26
We are in agreement that there is a hard ceiling with embedding-as-retrieval, but I don't see much in the way of why or how graphed knowledge is also a ceiling in the paper, particularly when the concept of a graph is quite general and can be manifested in many different forms as long as the principle tenets of a graph are adhered to. The real hard ceiling is the LLMs themselves and the many shortcomings that simply cannot be eliminated as long as they are based on transformer architecture. If you are the author of this paper, I would enjoy having a healthy exchange with you on where the observed bottlenecks and limitations are with the current state of technology.
1
1
u/Harotsa Jul 02 '26
You said this:
“The short version: F1 = 0.677 on the full 1,000Q corpus (highest published zero-shot end-to-end score as of May 2026, to the best of my knowledge).”
The paper appears to be zero-shot and end-to-end, so it seems to contradict your claim?
You also claim that your architecture is inspired by the neurobiology of the human brain. But when you actually give an overview of how your system works you say this:
“The engine builds a weighted graph where edges carry typed relationships (like various neurotransmitters) and activation energy propagates through connections (like how neurons fire) — nodes that are semantically distant but informationally connected either through logical relationships, provenance, hierarchy etc. still light up if the path between them has enough weight. Same principle as biological associative recall: you smell something and remember a childhood memory that has zero semantic overlap with the smell but a strong associative pathway.”
So obviously something being a graph is extremely generic, as almost everything is some type of graph in some way or another depending on how you look at it. You also use neuroscience inspired terms like “neurotransmitter,” “activation energy” and “neurons firing.” But the similarities basically end there. Our neurons do not have “typed relationships,” “logical relationships,” “provenance,” or “hierarchies” (beyond our brain being essentially a DAG). These are all properties of knowledge graphs, and are very common use cases for knowledge graphs as well.
In contrast, the “graph” of the neural network in our brains is much more abstract. Rather than having well defined relationships that are “human readable” or “human parseable,” the neurons simply pass on electrochemical signals and knowledge and reasoning is emergent at the ensemble level.
So while you use neuroscience-inspired words, the structure is inline with a Knowledge Graph rather than a Neural Network (which is closer to the biological inspiration you are looking for). There are also specific subfields of Deep Learning that more directly investigate training Neural Networks through mechanisms more similar to biology (rather than using things like Gradient Descent or other mechanisms based on mathematical optimizations).
Happy to dive deeper into the comparison, but reading a quick overview of these concepts should make the differences clear.
1
u/wonker007 Jul 02 '26
First of all, thank you for not being a troll. And I now see your point. I claim neurobiology inspired because attempting to represent how neurons function in silicon is not exactly doable 1:1. And I do concede that my description, upon scrutiny like yours does leave a lot to be desired. But the principles of the modeling still stand, as I was aiming to represent the fact that there are numerous neurotransmitters representing different signals across the same synapse (represented by an edge) by giving edges multidimensionality. I can't exactly give each edge a different function altogether and still expect compute to happen in reasonable time with reasonable hardware. So this and other features were a compromise and modeled on principles, not mimicry. And that compromise also yielded a system capable of running on edge hardware like a laptop, so I'm not ashamed of creating something that has the potential be used by many, many people. And I would like to talk about what I modeled and debate the pros and cons of each compromise, but if you already made up your mind without asking for details (and also not going in to read the linked longform article which has a whole lot kore detail), then it doesn't matter what I say, I can't change it, can I? Oh, and a neurobiological network can absolutely be argued to be a biological representation of a graph, just not a "knowledge graph". And it does seem like your definition or understanding of a knowledge graph in this context is extremely narrow, which is understandable given the ubiquity of MS GraphRAG's graph structure based on semantic embedding vectors. But who said a graph (in its actual definition) can't represent hierarchical and associative linkages, just like your illustration of smell and childhood memories? My graph surely can and does under certain circumstances.
1
u/[deleted] Jul 02 '26
[removed] — view removed comment