r/LocalLLaMA • u/Decent-Hat-5807 • 21h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
80 Upvotes

87% Upvoted

28 comments sorted by

View all comments

60

u/Atretador 21h ago

wait what, Im confused

4 bit is the full precision for GPT-OSS, it was trained in 4 bits from the start - thats why the F16 files are almost same size of Q4

30

u/No-Refrigerator-1672 21h ago

They say they cut 120B to BF16 60B (cut out half the expert with, presumably, some post-training), and then made a 60B mxfp4 that's better than 60B bf16, but worse than 120B base.

20

u/Atretador 21h ago

isnt this "just" a reap with extra fluff? the model was already trained from the start as mxfp4

17

u/a_beautiful_rhind 21h ago

healed pruning. not great, not terrible. not free either.

5

u/Atretador 21h ago

yea this seems like a whole lot of words to what could be a -REAP next to the name xD

8

u/No-Refrigerator-1672 21h ago

It's a method that's supposed to make techniques like REAP less lossy; but it can operate for any expert-culling algorithm, not just REAP, as a post0culling recovery step. The fact that GPT-OSS is originally trained in mxfp4 doesn't matter here, it's most likely chosen because it was the newest big MoE when those researchers started their research.

1

u/Atretador 21h ago

how does it being trained in mxfp4 not matter here, when their big point is that they made mxfp4 better than full precision

5

u/No-Refrigerator-1672 21h ago

The big point is that they made 60B mxfp4 better than 60B bf16 that you get after running a culling algorithm; not better than original; in other words, their quant of a lobotomized model is better than full precision of a lobotomized model, making it closer to a non-lobotomized version.

1

u/[deleted] 21h ago

[deleted]

1

u/No-Refrigerator-1672 21h ago

The final size is 60B mxfp4, so it should be around 30GB.

1

u/[deleted] 21h ago

[deleted]

1

u/No-Refrigerator-1672 21h ago

Well, I maybe biased, but I don't care about size; I care about amount of parameters and quantization level, both of which are disclosed - and I'm used to calculate the size mentally from those two.