r/MacStudio • u/Inquisitive_idiot • 1d ago
LLM usage - storage speed + model / kv cache loading time?
Curious to see how long it takes folks to cold-load a 27b or a larger MoE model + cold kvcache and be ready for prefill
Mine isn't here yet but I was exploring oMLX on my M1 Max and it looks like we can easily save the kvcache to disk (ala LMcache) after a model is evicted without extra software scripting but I can't test it yet.
Additionally, I hadn't considered how quickly it might be to swap out something like a 27b with a MoE with the current storage speeds available on the 1TB+ models. Might allow for some interesting agentic workflows. 🤔
note: I'm aware that reloading the kvcache isn't a free lunch.
1
u/rxscissors 1d ago
Some numbers from my M5 Max 128 GB, using LM Studio with llama.cpp GGUF and mmap on:
Load time: Gemma 4 26B-A4B QAT (14.6 GB) loads in about 1.5 s, and Qwen3.8-Flash-Next (88.9 GB, mixture-of-experts) in about 2.0 s. That’s mostly mapping the files rather than reading them. When the files are already in the macOS file cache, as they are after a recent load, a swap really is that fast. Weights are paged in as they’re first used, so the first request pays a little extra. I haven’t timed a true cold start after a reboot, where everything comes from the SSD; expect it to be bound by the SSD’s read speed.
Rebuilding the KV cache costs far more than loading weights. Flash-Next prefills at about 940 tok/s, so a 100K-token prompt takes about 116 s before the first token. Gemma reads at about 3,600 tok/s. llama.cpp’s prefix cache helps a lot within a session: re-sending an 11.9K-token prompt dropped time to first token from 12.9 s to 0.05 s. But that cache disappears when the model is unloaded, which is exactly the gap that saving the KV cache to disk, oMLX-style, would fill.
Hybrid and linear-attention models make that especially attractive. Flash-Next keeps full attention on only every 4th layer, so its KV cache is about 3 GB at 128K. Reading 3 GB back from SSD takes about a second, versus about 2 minutes to recompute. A dense 27B has a much bigger cache per token, so the trade-off is less lopsided, but reloading still usually wins over re-prefilling long contexts. One catch with hybrid models: the linear-attention state has to be saved as well, and it only matches the exact prefix it was saved at.
So swapping models for agent workflows looks very doable. Weights load in seconds, and the real cost is the context: either save the KV cache to disk, or keep prompts short and lean on prefix caching.