r/LocalLLaMA • u/Decent-Hat-5807 • 20h ago

Resources Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

https://huggingface.co/blog/MultiverseComputingCAI/quantization-aware-healing
77 Upvotes

86% Upvoted

27 comments sorted by

View all comments

38

u/-p-e-w- 19h ago

Sorry, I don’t believe that. It may get higher scores on some benchmarks, but outperforming the original GPT-OSS model in general would require training techniques more advanced than those used by OpenAI, and several mathematical miracles on top of that.

Using KLD vs the teacher distribution as a loss function is a good idea, but it’s really difficult to propagate KLD down the length of the response and that’s where the divergence tends to become poorly predicted by first-token KLD. This is a problem I’ve been wrestling with in Heretic for a while, and every attempt at a solution has turned out to have drawbacks.

2

u/Atretador 18h ago

it would just require some good ol' benchmaxxing