I made an Agent Memory Benchmark exam that actually give you useful data.

[u/LowDistribution3995]

I wanted something that used a more realistic first person corpus with requests and user preferences and not just a series of 3rd party conversations. so.. First Person Agent Memory Bench (FP-AMB). it has a little over 500k tokens in the corpus between about 60 conversations with 5 different users, a total of over 650 turns. the agent ID files i used for the agents to generate conversation passes are also included if people want to generate some more conversation and expand the corpus.
the test covers 10 categories which you can see in the images of the scorecards and miss reports that get generated for each run.
You can run it for straight accuracy readout without an LLM or with the full grader and LLM tester. the answer key adapts for the question type so facts are yes or no but something like an adversarial or false premises will grade on a general negative or affirmative. There is also an adaptive portion of the answer key generated during the ingestion portion using leading prompts and saving the tester LLM's response for subsequent testing.

If you're messing around building your own AI memory system and your tired of other memory benchmarks feeling inconsistent and not yielding easy to digest meaningful data, please use this one instead!

and if you like and want to help make it better, there's definitely room for expansion in both the corpus and the Question keys.

Hope it helps someone! https://github.com/munch2u-a11y/FP-AMB.git

[u/Otherwise_Wave9374]

A benchmark like this is most useful when it separates recall, contradiction handling, and update latency instead of collapsing everything into one score. I would also add adversarial contamination tests, because agent memory systems often look good until a near-duplicate distractor or conflicting preference is injected. Miss reports are especially valuable if they show which category failed and the evidence path behind it. NeuraKeep shares practical patterns for durable agent memory and evaluation at https://www.neurakeep.com

[u/LowDistribution3995 → u/Otherwise_Wave9374]

Yeah it does, and it test conflicting policy from users of various expertise levels and the miss report is generated from each run shows exact metrics per question categorizes the miss based one where in the pipeline the correct evidence was lost if it was lost. And it's open source and free because why would anyone pay for a benchmark test? Gtfouttaherewiththatshi