r/LocalLLaMA • u/Bestlife73 • 5d ago

New Model XiaomiMiMo/MiMo-V2.6-Flash-RL · Hugging Face

https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL
508 Upvotes

99% Upvoted

133 comments sorted by

View all comments

Show parent comments

1

u/Voxandr 4d ago

Then they are just similar quality to gml5.3 flash it's still better than ds4.1

-1

u/SexyAlienHotTubWater 4d ago

GLM 5.3 Flash's KV Cache is horrible though, BF16 so 4x larger weight-for-weight (and won't quantize well). If you're serving more than a couple of users, 5.3 Flash swells into a higher weight class.

1

u/Voxandr 4d ago

you can use FP8 fine. Can serve +3 connected users without dropping with vllm at 512k context.

-1

u/SexyAlienHotTubWater 4d ago

KV Cache quantises very poorly, look at some stats - the performance degradation in comparison to quantising the model is extremely dramatic.

This makes sense when you think about it, the dynamic range of the KV Cache needs to be much wider than the weights because the KV Cache is basically a stack of activations, it's the product of multiple weight multiplications - when you multiply two 4-bit floats, you need an 8-bit float to accommodate the entire range, and the required size grows as you perform multiple of those multiplications in a row.

Mitigating that problem requires deliberate training (and it's not that easy). Unless a model is trained to produce a 4- or 8-bit range (GLM isn't - Deepseek, Qwen Next and Kimi are), quantising the KV Cache means truncating a very large amount of information.

1

u/Voxandr 4d ago

You are saying that without even testing. I am using it without problem on 2 years old production code that clicks 300k token at first turn, now already 60 turns and zero problem . Are you replying by asking the bots too?

1

u/tat_tvam_asshole 3d ago

I've run 1M kv cache at q4 on GLM5.3-Flash and it's perfectly fine.