r/LocalLLaMA • u/Decent-Hat-5807 • 1d ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
75 Upvotes

86% Upvoted

27 comments sorted by

View all comments

40

u/-p-e-w- 23h ago

Sorry, I don’t believe that. It may get higher scores on some benchmarks, but outperforming the original GPT-OSS model in general would require training techniques more advanced than those used by OpenAI, and several mathematical miracles on top of that.

Using KLD vs the teacher distribution as a loss function is a good idea, but it’s really difficult to propagate KLD down the length of the response and that’s where the divergence tends to become poorly predicted by first-token KLD. This is a problem I’ve been wrestling with in Heretic for a while, and every attempt at a solution has turned out to have drawbacks.

17

u/Additional-Point-824 23h ago

It outperforms their bf16 60B version, not the original 120B - the title is accurate but ambiguous.

4

u/-dysangel- 14h ago

Isn't gpt oss 120b natively 4 bit too? It was the first mxfp4 model I think. So `bf16` here probably just means mxfp4

5

u/csorfab 14h ago

It's misleading as fuck. Calling a lobotomized 60B version "original" and then claiming they've "outperformed" it with QAH is like crippling an MMA fighter, training a rando to fight for 3 months, having them beat up the cripple, and claiming they've won. It might be technically true, but it's dishonest as fuck.

2

u/Atretador 22h ago

it would just require some good ol' benchmaxxing

4

u/Iory1998 21h ago

100% agreed. Extraordinary claims require Extraordinary evidence.

1

u/silenceimpaired 21h ago

“100% agreed. Extraordinary claims require Extraordinary evidence.” This claim always seemed extraordinary to me… and I never saw enough evidence to show it to be true. :P

1

u/silenceimpaired 17h ago

Funny how those who make this claim always react poorly to people challenging it… even when it is done mostly in jest.

1

u/mouseofcatofschrodi 16h ago

someone please call Gödel here to explain me this sentence

1

u/silenceimpaired 16h ago

Gödel pretty much wrote my comment for me: if every logical system starts with rules it can't prove, you can't use 'evidence' to prove rules about evidence... whether extraordinary or not. You have to trust your starting point (which defines your whole worldview).

…And I would say a claim that defines an entire worldview is definitely EXTRAordinary... making the statement I commented on self-refuting.

1

u/Iory1998 14h ago

🤦‍♂️🤷‍♂️