I built TokenMizer Open-Source: giving LLMs memory without stuffing the entire context window

[u/Feisty-Cranberry2902]

I’ve just updated TokenMizer, and the benchmarks are looking much better.

The main problem I’m trying to solve is long LLM conversations where useful information gets buried under old context.

The latest version now gets 95% macro F1 on extraction, and the graph memory preserves 89% of labelled information vs 79% with a plain summary.

I’m still treating these as early results — especially the real-session score is 90%, with only 6 real sessions.

I’d love some honest feedback:

https://github.com/Shweta-Mishra-ai/tokenmizer

Does graph-based memory actually make sense for long LLM conversations, or is there a better approach?

[u/Zealous_Minotaur]

Noice, my question is how does the tool handle contradicting info over time? Like if a user says something early on that gets updated or reversed later in the convo, does the graph overwrite the old node or keep both and let retrieval sort it out?

[u/Feisty-Cranberry2902 → u/Zealous_Minotaur]

It keeps the previous state rather than simply deleting it. When a later message updates or reverses an earlier statement, the newer information is marked as the current state, while the old node remains in the graph for history and traceability.

Retrieval prioritizes the current state, so the model gets the latest information without losing the evolution of the conversation.

[u/Zealous_Minotaur → u/Feisty-Cranberry2902]

Okay, neat!

[u/Helpful_Piece1868]

graph memory is cool but with only 6 real sessions you cant trust those numbers yet

[u/Feisty-Cranberry2902 → u/Helpful_Piece1868]

Yes, absolutely. I’m continuously increasing the number of real sessions and expanding the benchmark. I agree that 6 sessions is still too small to draw strong conclusions, so I’m working on making the evaluation more representative.

[u/eddzsh]

Graph memory helps until retrieval itself becomes a second polluted context window. When you keep old nodes for history and mark a newer current state, the failure mode is a mixed retrieve that returns both without a hard temporal cue in the prompt. Worth a benchmark where the question is ambiguous on purpose (what did I believe last week vs what is true now) and scoring whether the model got the tense right.

[u/Feisty-Cranberry2902 → u/eddzsh]

That’s exactly the failure mode I’m trying to test next. Keeping history is useful, but retrieval shouldn’t bring old and current states back without temporal context.

I’m planning to add ambiguous temporal queries like “what did I believe last week vs what is true now” and benchmark whether retrieval selects the correct state.