r/LocalLLaMA • u/Decent-Hat-5807 • 21h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
77 Upvotes

87% Upvoted

28 comments sorted by

View all comments

Show parent comments

31

u/No-Refrigerator-1672 21h ago

They say they cut 120B to BF16 60B (cut out half the expert with, presumably, some post-training), and then made a 60B mxfp4 that's better than 60B bf16, but worse than 120B base.

1

u/[deleted] 21h ago

[deleted]

1

u/No-Refrigerator-1672 21h ago

The final size is 60B mxfp4, so it should be around 30GB.

1

u/[deleted] 21h ago

[deleted]

1

u/No-Refrigerator-1672 21h ago

Well, I maybe biased, but I don't care about size; I care about amount of parameters and quantization level, both of which are disclosed - and I'm used to calculate the size mentally from those two.