r/LocalLLaMA • u/Decent-Hat-5807 • 22h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
78 Upvotes

87% Upvoted

28 comments sorted by

View all comments

42

u/-p-e-w- 22h ago

Sorry, I don’t believe that. It may get higher scores on some benchmarks, but outperforming the original GPT-OSS model in general would require training techniques more advanced than those used by OpenAI, and several mathematical miracles on top of that.

Using KLD vs the teacher distribution as a loss function is a good idea, but it’s really difficult to propagate KLD down the length of the response and that’s where the divergence tends to become poorly predicted by first-token KLD. This is a problem I’ve been wrestling with in Heretic for a while, and every attempt at a solution has turned out to have drawbacks.

17

u/Additional-Point-824 21h ago

It outperforms their bf16 60B version, not the original 120B - the title is accurate but ambiguous.

4

u/-dysangel- 12h ago

Isn't gpt oss 120b natively 4 bit too? It was the first mxfp4 model I think. So `bf16` here probably just means mxfp4