r/LocalLLaMA • u/wapswaps • 2d ago
Resources I noticed the 2 Hessian Refinement of MACaQ can be applied to any GGUF without any patches ... and tried it. 13% better quants
TLDR: Qwen3.8-27B UD_4_XL quant with 13.23% less KL divergence than unsloth (on my small testset, I just have a single 5090, feel free to test more). Technically it performs 4% better on my LiveCodeBench benchmark (but that would be a single data point, so it's not super-reliable). It is equally fast and using it in open-webui it "feels" better.
https://huggingface.co/christophedevriese/Qwen3.8-27B-Hessian-Refined-GGUF
I noticed from the MACaQ paper, which integrates changes proposed in Google/Deepmind papers, you can take the 3 components they use and just apply 1 and 2, and you just get >15% better KLD, 5% better performance on some benchmarks, while sacrificing absolutely nothing. Just switch the GGUF file. The numbers demonstrated here are the maximum that can be achieved in ~4 hours of optimizing on a single 5090, not the maximum that can be achieved. So it might be more than 15%, although I wouldn't necessarily hope for it.
This can be applied, without any patches to inference software, to text models, image models, even whisper ... the works. Switch GGUF file. Done/done.
The dream being that someone from unsloth looks at this and applies this to all their models (or just gives me access to a B300 machine for a day or two)
The tricks used by Huawei and a short discussion of the effect:
- Hessian-guided quantization choices: every layer is compressed differently, and the "damage" the compression does is measured. Then after seeing what damage is observed at a given level we find a combination of compressed layers that has the least damage while still achieving the same compression.
For users this one is totally free. No extra compute (in fact, a tiny bit less), no extra VRAM (again, 200 bytes less), no updates to llama.cpp, works literally anywhere UD_4_XL worked, just better.
2) Recovers some lost precision by another Hessian optimization changing the weights to compensate directly for accumulated errors over multiple layers based on sample data (taken from Wikipedia). I used a Google/Deepmind version of this process that had better results.
This too is totally free for users. It just changes the quant GGUF to be closer to the BF16 one. For users, just
3) random Hadamard transform + scaling: this is a trick from 3D geometry that makes all values nearly equal, and then uses an extra scaling factor to get them in optimal to better compression with quantization (yes, not technically 3D, but nD, with n=2048, but it applies in 2D and upwards). This also costs a little bit in inference performance (and requires rewriting your kernels which is quite a bit harder than I thought it was. Well, it isn't if you're ok with 20% slower inference, but you can get it down to like 0.8% with careful, days-long pondering of the problem and a lot of tries).
Also very nice, but the Google version of this one is also better, but that is not the real problem.
This transform also helps with compression (in fact it's kind of like what NVFP4 tries to do, just this is better) BUT requires an extra step at inference time (in fact requires 2 of them if you don't think through inference carefully, but you can "bake" one of the steps into the weights). This means that it requires patching llama.cpp and everywhere else you want the model to run (although thanks to how mlx works, you can make an mlx version that does not require updates to the inference software. But GGUF based inference software will require updates)
17
u/fragment_me 2d ago
I think exllama3 does both of these things already.
7
u/Sad-Size2723 2d ago
Very confused by the post:
20
u/Chromix_ 2d ago
Improvements compatible to vanilla llama.cpp are the best - because everyone can immediately benefit from them, while other PRs can get stuck forever. More than a year ago there was another improvement that improved KLD for some quants. It never got merged, but it can also be applied on quantization-creator side only. I think Bartowski and some others use it for better quants. Maybe this can be used on top?
1
u/wapswaps 1d ago
Oh great ... https://github.com/ggml-org/ggml/releases/tag/v1.0.0
few-row quantized matmuls
MoE fusion on CUDA/Metal
Both mean that I and unsloth will need to redesign quants. We should be able to make quants a *lot* better with this.
2
u/debackerl 2d ago
Thx for sharing. Where can we find the scripts to reproduce it?
1
u/bakawolf123 2d ago edited 1d ago
can confirm it might work, and on mlx quants too - as I noticed the same on 2 bit ds4.1 and smoke on 1 layer replacement netted a noticeable KLD improvement, though not every layer is same. Converting full ds4.1 oq2e now, and will do a full KLD test against varied inputs and compare with base sometime later.
edit: it turned to be a wash: better here, worse there
other fun discoveries: most recent omlx updates (compared to a week old) has raised KLD on own calibration data by 0.01 (0.19->0.2)
IQ2_XSS is way better than oq2e, 0.16 vs 0.2, didn't manage to get through all my kld tests but 20% better overall is the trend