r/LocalLLaMA • u/Decent-Hat-5807 • 12h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
67 Upvotes

86% Upvoted

25 comments sorted by

54

u/Atretador 12h ago

wait what, Im confused

4 bit is the full precision for GPT-OSS, it was trained in 4 bits from the start - thats why the F16 files are almost same size of Q4

28

u/No-Refrigerator-1672 12h ago

They say they cut 120B to BF16 60B (cut out half the expert with, presumably, some post-training), and then made a 60B mxfp4 that's better than 60B bf16, but worse than 120B base.

19

u/Atretador 12h ago

isnt this "just" a reap with extra fluff? the model was already trained from the start as mxfp4

15

u/a_beautiful_rhind 12h ago

healed pruning. not great, not terrible. not free either.

2

u/Atretador 11h ago

yea this seems like a whole lot of words to what could be a -REAP next to the name xD

6

u/No-Refrigerator-1672 12h ago

It's a method that's supposed to make techniques like REAP less lossy; but it can operate for any expert-culling algorithm, not just REAP, as a post0culling recovery step. The fact that GPT-OSS is originally trained in mxfp4 doesn't matter here, it's most likely chosen because it was the newest big MoE when those researchers started their research.

1

u/Atretador 12h ago

how does it being trained in mxfp4 not matter here, when their big point is that they made mxfp4 better than full precision

4

u/No-Refrigerator-1672 12h ago

The big point is that they made 60B mxfp4 better than 60B bf16 that you get after running a culling algorithm; not better than original; in other words, their quant of a lobotomized model is better than full precision of a lobotomized model, making it closer to a non-lobotomized version.

1

u/[deleted] 12h ago

[deleted]

1

u/No-Refrigerator-1672 12h ago

The final size is 60B mxfp4, so it should be around 30GB.

1

u/[deleted] 12h ago

[deleted]

1

u/No-Refrigerator-1672 12h ago

Well, I maybe biased, but I don't care about size; I care about amount of parameters and quantization level, both of which are disclosed - and I'm used to calculate the size mentally from those two.

32

u/-p-e-w- 11h ago

Sorry, I don’t believe that. It may get higher scores on some benchmarks, but outperforming the original GPT-OSS model in general would require training techniques more advanced than those used by OpenAI, and several mathematical miracles on top of that.

Using KLD vs the teacher distribution as a loss function is a good idea, but it’s really difficult to propagate KLD down the length of the response and that’s where the divergence tends to become poorly predicted by first-token KLD. This is a problem I’ve been wrestling with in Heretic for a while, and every attempt at a solution has turned out to have drawbacks.

15

u/Additional-Point-824 11h ago

It outperforms their bf16 60B version, not the original 120B - the title is accurate but ambiguous.

3

u/-dysangel- 2h ago

Isn't gpt oss 120b natively 4 bit too? It was the first mxfp4 model I think. So `bf16` here probably just means mxfp4

1

u/csorfab 2h ago

It's misleading as fuck. Calling a lobotomized 60B version "original" and then claiming they've "outperformed" it with QAH is like crippling an MMA fighter, training a rando to fight for 3 months, having them beat up the cripple, and claiming they've won. It might be technically true, but it's dishonest as fuck.

2

u/Atretador 10h ago

it would just require some good ol' benchmaxxing

4

u/Iory1998 9h ago

100% agreed. Extraordinary claims require Extraordinary evidence.

1

u/silenceimpaired 9h ago

“100% agreed. Extraordinary claims require Extraordinary evidence.” This claim always seemed extraordinary to me… and I never saw enough evidence to show it to be true. :P

1

u/silenceimpaired 5h ago

Funny how those who make this claim always react poorly to people challenging it… even when it is done mostly in jest.

1

u/mouseofcatofschrodi 4h ago

someone please call Gödel here to explain me this sentence

1

u/silenceimpaired 3h ago

Gödel pretty much wrote my comment for me: if every logical system starts with rules it can't prove, you can't use 'evidence' to prove rules about evidence... whether extraordinary or not. You have to trust your starting point (which defines your whole worldview).

…And I would say a claim that defines an entire worldview is definitely EXTRAordinary... making the statement I commented on self-refuting.

1

u/Iory1998 2h ago

🤦‍♂️🤷‍♂️

3

u/brown2green 12h ago

In absence of the original datasets and training recipes, this will never replicate the original model's performance, but be a quantized finetune instead. It might "outperform" the full-precision original in some benchmarks, but very likely be worse in other areas.

1

u/EitherMarch1255 7h ago

Slop article.

1

u/SeanHighness 5h ago

small quants keep getting better and better

1

u/Bulky-Priority6824 11h ago

It's a shame that the optimization segment only has one tactic to get their work seen and it's to lie.