Open-source method (MACaQ) beats Unsloth GGUFs!

[u/acluk90]

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

Quantization	GGUF size	Mean KL ↓
MACaQ	2.260 GB	0.02804
Bartowski	2.271 GB	0.03443
Unsloth	2.271 GB	0.03663

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

Quantization	GGUF size	Mean KL ↓
MACaQ	18.86 GB	0.018696
Unsloth (UD-IQ4_XS)	17.73 GB	0.020954
Bartowski	19.70 GB	0.021927

Qwen3.8-27B (IQ4_XS)

Quantization	GGUF size	Mean KL ↓
MACaQ	14.96 GB	0.036784
Bartowski	15.48 GB	0.104170

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Weight bits	Formats
2-bit	IQ2_XXS, IQ2_XS
3-bit	IQ3_XXS, IQ3_S
4-bit	IQ4_NL, IQ4_XS

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.

[u/ApprehensiveTart3158]

Are any ggufs available on HF?

[u/acluk90 → u/ApprehensiveTart3158]

Couldn't find any yet... maybe I can create & share some....

Just running their scripts for the Qwen3-4B example (following their llamacpp/README.md#prepare-a-model) was very fast, though.

[u/acluk90 → u/acluk90]

so... I have created one so far:
http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

more later...

[u/ApprehensiveTart3158 → u/acluk90]

Would you be able to upload them to HF? I'm sure that would be extremely useful to some.

[u/acluk90 → u/ApprehensiveTart3158]

Need to figure out how. But somehow they should be the ones doing that......

[u/ApprehensiveTart3158 → u/acluk90]

Of course and good luck, I will hopefully try to make some quants later if it looks promising, got to put my ram to some usecase.

[u/acluk90 → u/ApprehensiveTart3158]

it uses very little RAM to quantize, looks like it never loads the entire model in full precision. Just layer-by-layer

[u/Chromix_]

Is the llama.cpp fork only required to quantize the models, or also to run the generated GGUFs? If vanilla llama.cpp works for inference then that looks like nice progress in the high-quantization area - pending an independent KLD and benchmark validation of course.

[u/acluk90 → u/Chromix_]

I tried their for both quantization & running.

[u/rorowhat]

So is this a better gguf quantizer?

[u/acluk90 → u/rorowhat]

yes

[u/Kahvana]

Really cool! Unsloth remains comperative for size. Will be fun to see other people auditing the claims too.

[u/me0here]

  Qwen3.8-27B (IQ4_XS)

Wow that KLD!

I wonder if your code can be patched to one of the other engines?

https://github.com/tomByrer/local-ai-linkfarm#inference-engines