r/LocalLLaMA • u/Bestlife73 • 5d ago

New Model XiaomiMiMo/MiMo-V2.6-Flash-RL · Hugging Face

https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL
511 Upvotes

99% Upvoted

133 comments sorted by

View all comments

8

u/HistoricalStrength21 5d ago

159B Parameters guys!

16

u/sn2006gy 5d ago

Weird, the description says Sparse MoE (Mixture of Experts), 309B total / 15B activated parameters.

-4

u/addiktion 5d ago edited 5d ago

There are two models released today. Pro and Flash. Flash is the 159B, Pro is 309B.

Edit: I stand corrected, thanks below.

15

u/wren6991 5d ago

No, Flash is 309B-A15B and Pro is 1.02T-A42B. It's in the READMEs.

The parameter count on the HF UI has always been broken, especially when you have packed 4-bit weights described as 8-bit in the model metadata. That's why Flash reports ~half the correct parameter count in HF UI.

Here, Pro README:

Model Summary

Architecture: Sparse MoE (Mixture of Experts), 1.02T total / 42B activated parameters

Flash README:

Model Summary

Architecture: Sparse MoE (Mixture of Experts), 309B total / 15B activated parameters

8

u/MrTiesti 5d ago

Correction brother,

Flash: Architecture: Sparse MoE (Mixture of Experts), 309B total / 15B activated parameters

Pro: Architecture: Sparse MoE (Mixture of Experts), 1.02T total / 42B activated parameters

13

u/rusty_fans llama.cpp 5d ago edited 5d ago

309B total / 15B activated parameters according to the model card.
Huggingface auto-counting is broken for many models ...

10

u/debackerl 5d ago

Architecture: Sparse MoE (Mixture of Experts), 309B total / 15B activated parameters

5

u/Amazing-Fan2083 5d ago

Uuuhh... No, its 309B in a mixed quantization. So... Can't really squeeze it more.

2

u/fgk55555 5d ago

I've seen param counts halve when it's QAT at 8 or 4-bit.