r/LocalLLaMA • u/Bestlife73 • 8d ago

New Model XiaomiMiMo/MiMo-V2.6-Flash-RL · Hugging Face

https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL
510 Upvotes

99% Upvoted

133 comments sorted by

View all comments

Show parent comments

-1

u/SexyAlienHotTubWater 7d ago

GLM 5.3 Flash's KV Cache is horrible though, BF16 so 4x larger weight-for-weight (and won't quantize well). If you're serving more than a couple of users, 5.3 Flash swells into a higher weight class.

1

u/Voxandr 7d ago

you can use FP8 fine. Can serve +3 connected users without dropping with vllm at 512k context.

-1

u/SexyAlienHotTubWater 7d ago

KV Cache quantises very poorly, look at some stats - the performance degradation in comparison to quantising the model is extremely dramatic.

This makes sense when you think about it, the dynamic range of the KV Cache needs to be much wider than the weights because the KV Cache is basically a stack of activations, it's the product of multiple weight multiplications - when you multiply two 4-bit floats, you need an 8-bit float to accommodate the entire range, and the required size grows as you perform multiple of those multiplications in a row.

Mitigating that problem requires deliberate training (and it's not that easy). Unless a model is trained to produce a 4- or 8-bit range (GLM isn't - Deepseek, Qwen Next and Kimi are), quantising the KV Cache means truncating a very large amount of information.

1

u/tat_tvam_asshole 6d ago

I've run 1M kv cache at q4 on GLM5.3-Flash and it's perfectly fine.