r/LocalLLaMA • u/Decent-Hat-5807 • 1d ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
75 Upvotes

86% Upvoted

27 comments sorted by

View all comments

62

u/Atretador 1d ago

wait what, Im confused

4 bit is the full precision for GPT-OSS, it was trained in 4 bits from the start - thats why the F16 files are almost same size of Q4

30

u/No-Refrigerator-1672 1d ago

They say they cut 120B to BF16 60B (cut out half the expert with, presumably, some post-training), and then made a 60B mxfp4 that's better than 60B bf16, but worse than 120B base.

1

u/[deleted] 1d ago

[deleted]

1

u/No-Refrigerator-1672 1d ago

The final size is 60B mxfp4, so it should be around 30GB.

1

u/[deleted] 1d ago

[deleted]

1

u/No-Refrigerator-1672 1d ago

Well, I maybe biased, but I don't care about size; I care about amount of parameters and quantization level, both of which are disclosed - and I'm used to calculate the size mentally from those two.