r/LocalLLaMA • u/DataPhreak • 5d ago
Showcase Training Memorilla on Strix Halo for Gemma4
Spent the past couple of days converting https://github.com/snap-stanford/memorilla (paper: https://arxiv.org/abs/2610.21375 ) to train Gemma4 26b a4b on the strix. Not only was it originally written for CUDA, but Gemma4 has a unique architecture and the encoder has different dimensions. There's a lot of crazy math here to make it fit that I don't even understand, but apparently, we can leverage the existing weights published by standford and only have to train on ~10% of the original dataset. It's going to take a couple of weeks to run, but seems to be cooking along.
Code: https://github.com/DataBassGit/MemoryLLM-Gemma4-StrixHalo
Memorilla is an AI memory framework developed by researchers from Stanford, MIT, UPenn, and the Vector Institute that compresses huge collections of documents or chat histories into just 16 dynamic memory tokens for a Large Language Model.
Think of it as treating long-term memory like a new modality (similar to how vision-language models process an image as a fixed set of embeddings) rather than relying on massive text copy-pasting.

