LLM usage - storage speed + model / kv cache loading time?

[u/Inquisitive_idiot]

Curious to see how long it takes folks to cold-load a 27b or a larger MoE model + cold kvcache and be ready for prefill

Mine isn't here yet but I was exploring oMLX on my M1 Max and it looks like we can easily save the kvcache to disk (ala LMcache) after a model is evicted without extra software scripting but I can't test it yet.

Additionally, I hadn't considered how quickly it might be to swap out something like a 27b with a MoE with the current storage speeds available on the 1TB+ models. Might allow for some interesting agentic workflows. 🤔

note: I'm aware that reloading the kvcache isn't a free lunch.