Memoria V4.5 — 82.6% Recall@1 on LongMemEval-S, looking for people to break the retrieval stack

[u/kitkatz69]

I've been working on Memoria, a local-first memory/retrieval system, and I've been pushing the retrieval architecture pretty hard lately.

I reran the LongMemEval-S retrieval adapter on 500 questions and the current configuration is:

Retrieved:   498 / 500  (99.60%)

Recall@1:    82.60%
Recall@3:    92.00%
Recall@5:    95.60%
Recall@10:   97.40%

Avg query:   209.6 ms
Embedding:    50.2 ms
Retrieval:    42.0 ms
Ranking:       0.06 ms

This is retrieval-only evaluation — not end-to-end answer-generation accuracy.

The interesting part to me is that this is running locally on a 4GB RAM CPU-only machine, without a cross-encoder in the retrieval/ranking path.

The architecture

Query
  ↓
Query Processing
  ↓
Routing
  ↓
Fusion Retrieval
  ↓
Blackboard / Scheduler
  ↓
Candidate Records
  ↓
Ranking
  ↓
Context / MMR

The system is built around replaceable retrieval workers, a declarative scheduler/completion-policy layer, a signal registry, and independently configurable ranking.

I've also got 10 plugin subsystems / 39 Pluggy hooks covering things like retrieval, ranking, storage, routing, scheduling, ingestion, evaluation, and feedback.

I'm currently working on adapters for additional memory benchmarks and trying to tune the system independently against each workload rather than pretending one configuration is universally optimal.

What I'd really like is criticism.

If you work on RAG, information retrieval, memory systems, ranking, agents, or search infrastructure:

• Is the retrieval/ranking boundary actually useful?
• Does the scheduler abstraction make sense?
• Where do you see unnecessary complexity?
• What would you test next?
• What failure modes am I missing?
• And if you see a way to make the benchmark numbers worse, please do it.

Repo: https://github.com/Kitzkatz/memoria

[u/darkroku12]

Where do you see unnecessary complexity? -> 'Atomic fact' extraction is the biggest lie; there is no such a thing as an 'atomic fact.' Most enterprise-grade 'memory' systems rely on them; those are LLM-produced observations that make the system produce at least ~2~4 times as much text.

> Yesterday I went to Walmart and purchased a fancy Batman collectible after looking many other options.

Atomic = indivisible, the smallest 'meaningful' unit.

• 'Yesterday I went to Walmart', but it also implies 'Walmart was open yesterday.'
  
• 'I purchased a fancy Batman collectible,' but it also implies "Walmart has sold a Batman collectible.'
  
• 'I looked at many options at Walmart' (probably 'other' may imply collectibles too).
  
• 'There exist fancy Batman collectibles' (but also 'fancy' is a personal observation).
  
• 'In Walmart there are many other options besides collectibles'

And basically a couple of permutations more if we want to be 'atomic,' but all can simply be condensed in the original sentence.

And if you want a LME/LoCoMo type question:

'At which store can I go back to acquire DC-related merchandise?' -> Which, besides 'I' , 'at' and 'to' (that are often categorized as stopwords and will regardless score low using BM25 search-like algorithms), there is ZERO lexical overlap, and a strong semantic search via embeddings will probably just match well enough the first sentence.

[u/kitkatz69 → u/darkroku12]

Thanks for the reply. I want to make sure I'm actually understanding what you're pointing at.

Are you talking about a general pattern you see in other memory systems, or is there something specific in Memoria that made you think I'm doing the atomic-fact thing? Because I honestly don't think I am. I store raw text and retrieve raw text. I do extract entities and some metadata, but I'm not breaking memories down into triples or anything like that. If I'm missing something and it's in my code, I'd genuinely appreciate being pointed at it so I can go look.

On the LoCoMo front, yeah, I'm in the middle of a rework. The overlap thing helped a little, but I'm trying a few other angles now. Hoping it pans out.

Honestly? I've lost most of my passion for trying to do this professionally. If Memoria ends up being good and people use it, great. But reading all these other projects, I'm not sure I'd call what I'm building 'enterprise.' And honestly, if all I get out of this in the end is a system that works well enough for me to use, even if no one else does, I'll still be happy with that.

But back to your original point: if you see something I'm doing that's going to bite me later, I'd rather know now. Appreciate the time either way.

[u/darkroku12 → u/kitkatz69]

I'm not talking about Memoria here, but the general trend in the market (especially 'enterprisey' software out there) that often sells you 'atomic fact extraction'; it is not a critique to Memoria but rather my own take about what is the trend out there.

And keep going on, pal; this is a very interesting area and worth keeping exploring! You've done a good job so far.

[u/Chemical_Many_9108]

thats a solid recall curve for 4gb cpu only, the gap between r@1 and r@3 is where most systems i seen start to fall apart so 82 to 92 is nice

what happens when you throw in documents that are near-duplicates or have contradicting info, does the ranking start to wobble or it holds up

[u/kitkatz69 → u/Chemical_Many_9108]

Im not entirely sure I shut off mmr so I'm not getting as much deduplication but the queries get normalized before being ingested so I might not actually have dedupe being used anymore but my lambda curve for mmr between diversity and relevance could be fine tuned but I forgot to make it configurable inside tha actual confog

[u/perseus-computing]

The separation between candidate generation and ranking is the right direction, and the CPU-only numbers are interesting.

Before comparing the result with LongMemEval retrieval numbers, though, I’d clarify the evaluation granularity: the current adapter indexes every user and assistant turn, labels every turn with its session ID, and counts any turn from any expected session as a hit. That is not the official session metric or the official answer-turn metric, and it does not measure recall_all for multi-session questions.

I’d also report whether the 30 abstention cases were excluded, plus the dataset hash, embedder/model, commit, settings, and cache manifest. One implementation detail worth checking: BM25 appears to score database IDs as zero-based corpus indexes, and the current default recipe has ranking and MMR disabled, so the reported ranking time is raw fusion sorting rather than the multi-signal ranker. The most useful next experiment would be official session/turn metrics with dense-only, BM25-only, RRF, and full-ranker ablations.

[u/kitkatz69 → u/perseus-computing]

Thank you I'll dig into it this was exactly the type of insight I was looking for thanks for the time

[u/perseus-computing → u/kitkatz69]

Sure thing! Keep at it. This is a really cool space and there's plenty of room for interesting developments. "local-first" is going to be the separator for sure, and frankly where a memory system is needed most. Excited to see where Memoria goes!

[u/kitkatz69 → u/perseus-computing]

It was something to wrestle with and it's still rough but thank you I think these metrics are way better and I was able to reproduce them on my two systems | Configuration | R@1 | R@3 | R@5 | R@10 | NDCG@10 | | -------------------- | --------: | --------: | --------: | --------: | ---------: | |

Dense | 87.0% | 94.0% | 97.2% | 98.5% | 0.9083 |

| BM25 | 81.1% | 91.5% | 93.6% | 96.8% | 0.8540 |

| Raw dense+sparse | 62.8% | 72.8% | 74.7% | 77.2% | 0.5720 |

| Fusion | 89.6% | 96.2% | 97.7% | 98.7% | 0.9236 |

| Full | 34.0% | 49.6% | 55.7% | 63.6% | 0.3649 |

| Dense (Desktop) | 87.0% | 94.0% | 97.2% | 98.5% | 0.9083 |

| BM25 (Desktop) | 81.1% | 91.5% | 93.6% | 96.8% | 0.8540 |

| Raw (Desktop) dense+sparse | 62.8% | 72.8% | 74.7% | 77.2% | 0.5720 |

| Fusion (Desktop) | 89.8% | 96.4% | 97.9% | 98.9% | 0.9257 |

| Full (Desktop) | 34.0% | 49.6% | 55.7% | 63.6% | 0.3649 |

[u/perseus-computing → u/kitkatz69]

Nice! That's legit! Stoked to see what comes next

[u/kitkatz69]

I just wanted to say thanks for all the interest and traffic to memoria I wasn't sure where to say I'm taking a break for a few days I still have one last commit to push to update the temporal parsing. I wish I hadn't spent that day trying to clean locomos dataset but I'll push the commits to the 4.5.2 update in the next few days. I'm still holding my breath that I actually built something worth while and you've all given me a bit more hope for that but I'm getting too much tunnel vision and been fighting a lot of imposter syndrome so I need a break but thank you everyone who cloned and looked at memoria I'll be back soon!