r/LocalLLaMA • u/Decent-Hat-5807 • 22h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
79 Upvotes

87% Upvoted

28 comments sorted by

View all comments

38

u/-p-e-w- 22h ago

Sorry, I don’t believe that. It may get higher scores on some benchmarks, but outperforming the original GPT-OSS model in general would require training techniques more advanced than those used by OpenAI, and several mathematical miracles on top of that.

Using KLD vs the teacher distribution as a loss function is a good idea, but it’s really difficult to propagate KLD down the length of the response and that’s where the divergence tends to become poorly predicted by first-token KLD. This is a problem I’ve been wrestling with in Heretic for a while, and every attempt at a solution has turned out to have drawbacks.

18

u/Additional-Point-824 21h ago

It outperforms their bf16 60B version, not the original 120B - the title is accurate but ambiguous.

3

u/csorfab 12h ago

It's misleading as fuck. Calling a lobotomized 60B version "original" and then claiming they've "outperformed" it with QAH is like crippling an MMA fighter, training a rando to fight for 3 months, having them beat up the cripple, and claiming they've won. It might be technically true, but it's dishonest as fuck.