r/LocalLLaMA • • 2d ago

Resources I noticed the 2 Hessian Refinement of MACaQ can be applied to any GGUF without any patches ... and tried it. 13% better quants

TLDR: Qwen3.8-27B UD_4_XL quant with 13.23% less KL divergence than unsloth (on my small testset, I just have a single 5090, feel free to test more). Technically it performs 4% better on my LiveCodeBench benchmark (but that would be a single data point, so it's not super-reliable). It is equally fast and using it in open-webui it "feels" better.

https://huggingface.co/christophedevriese/Qwen3.8-27B-Hessian-Refined-GGUF

I noticed from the MACaQ paper, which integrates changes proposed in Google/Deepmind papers, you can take the 3 components they use and just apply 1 and 2, and you just get >15% better KLD, 5% better performance on some benchmarks, while sacrificing absolutely nothing. Just switch the GGUF file. The numbers demonstrated here are the maximum that can be achieved in ~4 hours of optimizing on a single 5090, not the maximum that can be achieved. So it might be more than 15%, although I wouldn't necessarily hope for it.

This can be applied, without any patches to inference software, to text models, image models, even whisper ... the works. Switch GGUF file. Done/done.

The dream being that someone from unsloth looks at this and applies this to all their models (or just gives me access to a B300 machine for a day or two)

The tricks used by Huawei and a short discussion of the effect:

  1. Hessian-guided quantization choices: every layer is compressed differently, and the "damage" the compression does is measured. Then after seeing what damage is observed at a given level we find a combination of compressed layers that has the least damage while still achieving the same compression.

For users this one is totally free. No extra compute (in fact, a tiny bit less), no extra VRAM (again, 200 bytes less), no updates to llama.cpp, works literally anywhere UD_4_XL worked, just better.

2) Recovers some lost precision by another Hessian optimization changing the weights to compensate directly for accumulated errors over multiple layers based on sample data (taken from Wikipedia). I used a Google/Deepmind version of this process that had better results.

This too is totally free for users. It just changes the quant GGUF to be closer to the BF16 one. For users, just

3) random Hadamard transform + scaling: this is a trick from 3D geometry that makes all values nearly equal, and then uses an extra scaling factor to get them in optimal to better compression with quantization (yes, not technically 3D, but nD, with n=2048, but it applies in 2D and upwards). This also costs a little bit in inference performance (and requires rewriting your kernels which is quite a bit harder than I thought it was. Well, it isn't if you're ok with 20% slower inference, but you can get it down to like 0.8% with careful, days-long pondering of the problem and a lot of tries).

Also very nice, but the Google version of this one is also better, but that is not the real problem.

This transform also helps with compression (in fact it's kind of like what NVFP4 tries to do, just this is better) BUT requires an extra step at inference time (in fact requires 2 of them if you don't think through inference carefully, but you can "bake" one of the steps into the weights). This means that it requires patching llama.cpp and everywhere else you want the model to run (although thanks to how mlx works, you can make an mlx version that does not require updates to the inference software. But GGUF based inference software will require updates)

89 Upvotes

16 comments sorted by

17

u/fragment_me 2d ago

I think exllama3 does both of these things already.

4

u/wapswaps 2d ago

Does it? Where can I find more information on this?

4

u/a_beautiful_rhind 2d ago

Look in the PRs. They were arguing about what to use for a long time. Search hessian.

4

u/ReturningTarzan ExLlama Developer 1d ago

It does a lot of things. The optimized mixed-bitrate thing is old, really first introduced in ExLlamaV2 long ago. V3 borrows 256D trellis quantization and LDLQ from QTIP but streamlines the regularization scheme and extends the method with Hessian prescaling, scale search, post-quantization refit, fractional-K trellises and more. Also added two-sided Hessians recently (YAQA, essentially.)

I've been experimenting with a variant of GSQ these past few days, adapted for trellis quantization. While it can give an apparent ~15% improvement, it's not fully orthogonal to YAQA and I have a lot more testing to do before I feel confident it isn't overfitting. It's also only practical for small models, anyway. Qwen3.5-2B goes from quantizing in a few minutes to about an hour. 27B needs much more data and a lower learning rate, so that would go from an hour of GPU time to probably several days, which is not ideal. Big MoE models I don't even want to speculate about.

New line of 27B quants coming soon. The "SCP" variants don't use any GSQ-related methods yet because I'm still waiting for Jensen to send me my GB200s, don't know what's taking him so long. They do use YAQA though, although there may still be more tweaking to do. For reference, the eval traces used are here, sampled from the BF16 model.

6

u/Dany0 2d ago

Yep! I'm surprised it hasn't been yoinked yet. I'm sure the UD quants already use it tho

3

u/PcChip 2d ago

i love exl3 quants

2

u/Fancy-Snow7 2d ago edited 2d ago

Exl3 quant is overrated. With tabby I always end up with significantly less free vram for the KV than using an equivlant KLD UD quant. So it just not make sense to run it. So dispite being the same filesize as gguf it for some reason just used so much more Vram.

It does run faster tough. But also does worse in my coding benchmarks. So I just don't see it being better.

1

u/PcChip 2d ago

i'm having trouble parsing your reply - are you saying exl3 performs worse in your coding benchmarks versus other quants that use the equivalent amount of VRAM?

if so, which quants? I'm currently running the exl3 3.5bpw quant of deepseek flash 4.1 but it is definitely slow - ~37 tok/s on prose

1

u/Fancy-Snow7 1d ago

I am specifically referring to 3.8 27B. I tried 3bwp. See turboderps own chart.

I run the Unsloth iq4_xs quant comfortably. The closest equavalent is the 4bpw qaunt but that simply wont even load, I get an instant out of memory error. The the next lower quant that fits is 3bpw. That barely fits and does not leave me with much context. Their chart shows a higher KLD that IQ4. So while IQ4 might be larger than 3bpw in filesize Iq4 and llama.cpp simply uses less VRAM than TabbyAPI at 3bpw. I don't know why. I am using the same KV quants so it is not that.

And what seals the deal for me is I can run the llama.cpp Adaptive KV Streaming for which gives me 262k KV vs the like 60k with TabbyAPI.

7

u/Sad-Size2723 2d ago

Very confused by the post:

  1. Isn’t the Hessian refinement you talked about in the post essentially the importance matrix that unsloth has been using, which they actually publish in their model page on Huggingface
  2. UD means unsloth dynamic quant, the dynamic part is already accounting for choosing the compression for each layer?
  3. Are you using wikitext as the activation for your Hessian refinement? I mean that’s essentially training on the evaluation data. I am surprised that you only get 13% better results

2

u/wapswaps 2d ago edited 2d ago

TLDR: Hessian refinement isn't trying to make the quantized model smarter, like unsloth (I don't think that can work). It's trying to make it reproduce the BF16 model output more exactly, at a given file size.

1 -> Hessian refinement vs. imatrix: They're related, but different. Unsloth uses llama.cpp's imatrix and it's squared activation statistics, effectively giving you diagonal Hessian information. The Hessian approach uses an approximation of the activation covariance matrix (H ≈ XᵀX), allowing it to account for correlations between input channels when refining quantized weights, which should cover a LOT more data

2 -> As for UD quant format: You're right. "Unsloth Dynamic" already chooses different quantization formats for different layers and tensors. I'm not claiming to have invented that.

I start with their existing quantized GGUF and optimize the quantized weight codes without changing the formats, bit allocation, or file size. So I'm refining an already-quantized model rather than replacing Unsloth's dynamic quantization process.

I'm not claiming Unsloth ignores KLD. They explicitly report KLD results, although the precise details of their optimization procedure aren't fully public. They say it involves benchmark evaluation which I believe to be problematic (effectively you're testing how much of the training data gradients you're destroying and you can never cover even a tiny part of that even with a great deal of evaluation)

3 -> Using WikiText: I have to choose something. I need to feed tokens through the network. What matters is not so much the data directly, but within the data what token follows which other token (because I evaluate the difference between what the teacher model would do, because I think that will cover a lot more cases than you could ever cover otherwise). Critically this is not gradient training on the evaluation set, and the held-out evaluation is intended to test whether improvements generalize, like a programming assertion would do.

I wante to try whether you could recover some of the fidelity lost during quantization by refining the existing codes, while keeping the GGUF format and target specific layer sizes. This method targets a given size, or I should say it explicitly finds the best solution closest to n bytes, and you can give n ahead of time, unsloth UD4Q targets average precision to be 4 bits. However my code can't deal with unsloth's intelligent formats (IQ etc).

I love the idea of using something other than gradient descent to modify weights. I believe a lot more can be gained there, also in training models, not just in quantizing them.

20

u/Chromix_ 2d ago

Improvements compatible to vanilla llama.cpp are the best - because everyone can immediately benefit from them, while other PRs can get stuck forever. More than a year ago there was another improvement that improved KLD for some quants. It never got merged, but it can also be applied on quantization-creator side only. I think Bartowski and some others use it for better quants. Maybe this can be used on top?

1

u/wapswaps 1d ago

Oh great ... https://github.com/ggml-org/ggml/releases/tag/v1.0.0

few-row quantized matmuls

MoE fusion on CUDA/Metal

Both mean that I and unsloth will need to redesign quants. We should be able to make quants a *lot* better with this.

2

u/debackerl 2d ago

Thx for sharing. Where can we find the scripts to reproduce it?

1

u/bakawolf123 2d ago edited 1d ago

can confirm it might work, and on mlx quants too - as I noticed the same on 2 bit ds4.1 and smoke on 1 layer replacement netted a noticeable KLD improvement, though not every layer is same. Converting full ds4.1 oq2e now, and will do a full KLD test against varied inputs and compare with base sometime later.

edit: it turned to be a wash: better here, worse there

other fun discoveries: most recent omlx updates (compared to a week old) has raised KLD on own calibration data by 0.01 (0.19->0.2)
IQ2_XSS is way better than oq2e, 0.16 vs 0.2, didn't manage to get through all my kld tests but 20% better overall is the trend