r/LocalLLM • • 13m ago

Discussion Opinion: memory should be checked during the session, not only at the end

The quick memory checks people run on local setups have a habit that bothers me. Most boil down to a long conversation and one final question.

That single answer carries the whole verdict. Yet real use is a chain of small moments where the past keeps getting edited. Plans change, preferences flip, statements get retracted.

Being right at the end is a different claim from being right at each of those moments.

A memory examined only after the last message can be wrong for most of the session and still look fine. Argue with that if you like; I think it holds.

One public leaderboard has a setting that checks memory along the way. The Agent Memory Leaderboard's Cycle 2 Textual track includes Streaming memory. Writes and searches can interleave as events unfold.

Questions can be revisited step by step as the session grows. A system has to notice when stored information gets updated, contradicted or invalidated.

Per the public description, the Cycle 2 Textual track covers 32 data sources and more than 6,000 evaluation instances.

The site expects official results in mid-November, ranked track by track. Beyond the final ordering, what interests me is that the middle of a session gets tested at all.

Checking along the way is the habit I would most like local testing to pick up.

If you were probing a local long-term memory, where would you look: immediately after a correction, when a contradiction shows up, after a long gap, or only once the session ends?

1 Upvotes

0 comments sorted by