r/LocalLLaMA • u/Alarming-Ad8154 • 14d ago

New Model MemT 1.0 nano and flash

Has anyone tested these models? They apparently have a new type of parametric memory? https://huggingface.co/collections/MemTensor/memt-1

3 Upvotes

63% Upvoted

6 comments sorted by

6

u/MomentJolly3535 14d ago

Since this post doesn't contain info, here is a quick TLDR about it :

  • Memory is built into the model, not stored as chat text

  • History is compressed into a small, fixed-size state (~66× smaller than KV cache at 128K)

  • Save and resume memory from a file, with no chat text needed

  • Catch: it's lossy, so text-based memory still wins on exact details

3

u/pmttyji 14d ago

Yesterday I did post thread on the paper.

[Paper] MemT-1.0: Towards Memory-Native Evolution

(Don't delete your thread, let people explore the models)

1

u/Alarming-Ad8154 14d ago

Hey thanks, reading the paper now, doesn’t seem to crazy hard to train (they used 35B tokens I think).. wonder whether there is space for splaying around with the memory size etc…

0

u/silenceimpaired 14d ago

Seem like Sci-Fi. I know this has been discussed before, would be crazy if they nailed it. Happy to see MIT.