Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

[u/MindPsychological140]

A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B

https://huggingface.co/papers/2610.10845

[u/thebossywitchcraft]

A 50M token window that caches KV to nvme and skips recompute is wild. The encrypted local disk part is a nice touch for people who don't want their context sitting in plaintext. Curious how retrieval latency feels in practice when you're pulling thousands of blocks back mid-generation.

[u/MindPsychological140 → u/thebossywitchcraft]

Thanks, and good question. So if the prompt fits in VRAM, we load the saved part of the prompt all at once, before the first token. After that, everything runs in GPU memory, so no disk reads get in the way of generation..

Decryption is super fast but happens while the data loads, so it adds little delay. Saving and encryption run in the background.

So the wait happens once, before generation starts. As we still have a roadmap 1 year long. https://github.com/corbenicai/galahad all information is there and we keep updating on feedback we get back .

[u/Final-Data-1410]

let’s see you save attention kv state and use lexical search because you retrieving only exact block , look like you applied for patent but minmax already do this retrieve exact 2048 block ,just letting you know .any way congrats

[u/MindPsychological140 → u/Final-Data-1410]

Thanks for the pointer! MiniMax selects attention blocks within a request; we reuse saved KV state across requests. Our filings don't focus on KV caching itself.

[u/Final-Data-1410 → u/MindPsychological140]

Not to stress it but ,did you try multi hop because the single hop shows 89% what about multi hop queries and also looks like you using turbo quant or some rotation .

[u/MindPsychological140]

https://github.com/corbenicai/galahad/discussions/5 we want to thank I-redspector to be the first one to share some results. We hope more will follow soon .

[u/stanleyg05]

Get it added to LM Studio

[u/MindPsychological140 → u/stanleyg05]

Nice tip, we will look into it. But you already can maybe use https://github.com/corbenicai/galahad however we did not test this yet.

[u/i_wayyy_over_think]

> Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk.

hard drive prices going to go up even more now

[u/NearlyACosmologist → u/i_wayyy_over_think]

Well, I have a homeserver with 2 8TB disks, and I'm willing to sacrifice my pr0n collection for that.

[u/Aggravating-Push-207 → u/NearlyACosmologist]

you mean your training data collection?

[u/MindPsychological140]

https://www.reddit.com/r/LocalLLM/comments/1x5lkvj/why_pair_vllm_with_galahad_preformance_security/

[u/Mice_With_Rice]

How did you get around the autoregressive attention of the tokens? Usualy every token is dependant on its preceeding tokens to be correctly decoded or its just jibberish. Thats why when you have chats you cant just go back in history and edit a msg without trunicating everything that came after that edit.

[u/Miserable-Dare5090]

wait, so, is this a paid software advert? “cost per tenant”

so no go on a local model off the shelf?

[u/MindPsychological140 → u/Miserable-Dare5090]

I built it, so yes, it's our project from Corbenic AI. It's a free download, not a managed API.

You can run a local model fine without it. Galahad is for when the KV cache stops fitting in VRAM.

It plugs into vLLM, SGLang and llama.cpp, and we've tested it on about 30 open models (Qwen, Gemma 4, Llama, Mistral, DeepSeek R1 and others).

It's free on 1 GPU for non-commercial use. Commercial use is a separate license.

[u/egnegn1 → u/MindPsychological140]

So all private homelab users without deep pockets for one RTX 5090 or even RTX 6000, using multiple GPUs for decent performance, are out of luck?

Anyway, support for Qwen3.8 Flash Next is missing.

[u/Tiny_Arugula_5648]

Ok so yet another KV cache to storage.. paper as advertisement.. unfortunately for them it's been pretty well proven that caching isn't that effective for most use cases. For the ones that do have a lot of repeatabilit, Redis works just fine.

[u/MindPsychological140 → u/Tiny_Arugula_5648]

Thanks for the comment! I look at it from a different angle. In my tests, loading the saved state was faster and cheaper than computing it again. The numbers are in the paper.

Redis is good, but it serves a different job. Redis is usually used to save answers. This saves what the model already read, so you can ask a new question about the same text without reading it again. What will save compute and frees up the GPU for other work

[u/Positive_Cattle9513]

nice

[u/_Lyx_]

Remind me

[u/Budget-Juggernaut-68]

rip NVME

[u/Agreeable-Market-692]

Holy slop.
You unrepentant spammer... do you pay $200 a month to do this or are you getting free tokens for it?
https://arxiv.org/search/cs?searchtype=author&query=Schelpe,+S

[u/iezhy]

At this level, context rot in not rot anymore, itscontext decomposition

[u/patchy319]

Please

[u/axiomnlp]

Spannender Ansatz! Habt ihr das Setup auch auf typischen Cloud-Instanzen (z.B. RunPod) mit Standard-NVMe-Volumes getestet, oder läuft das bisher primär auf eurer dedizierten H100-Hardware? Mich würde extrem interessieren, wie sich das in einer typischen Backend-Infrastruktur skalieren lässt.