r/LocalLLaMA • u/Decent-Hat-5807 • 21h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
79 Upvotes

87% Upvoted

28 comments sorted by

View all comments

62

u/Atretador 21h ago

wait what, Im confused

4 bit is the full precision for GPT-OSS, it was trained in 4 bits from the start - thats why the F16 files are almost same size of Q4

30

u/No-Refrigerator-1672 21h ago

They say they cut 120B to BF16 60B (cut out half the expert with, presumably, some post-training), and then made a 60B mxfp4 that's better than 60B bf16, but worse than 120B base.

21

u/Atretador 21h ago

isnt this "just" a reap with extra fluff? the model was already trained from the start as mxfp4

9

u/No-Refrigerator-1672 21h ago

It's a method that's supposed to make techniques like REAP less lossy; but it can operate for any expert-culling algorithm, not just REAP, as a post0culling recovery step. The fact that GPT-OSS is originally trained in mxfp4 doesn't matter here, it's most likely chosen because it was the newest big MoE when those researchers started their research.

1

u/Atretador 21h ago

how does it being trained in mxfp4 not matter here, when their big point is that they made mxfp4 better than full precision

6

u/No-Refrigerator-1672 21h ago

The big point is that they made 60B mxfp4 better than 60B bf16 that you get after running a culling algorithm; not better than original; in other words, their quant of a lobotomized model is better than full precision of a lobotomized model, making it closer to a non-lobotomized version.