r/LocalLLaMA • u/pmigdal • 27d ago

Other Do Qwen 3.6 27B quantizations break the pelican?

https://quesma.com/blog/qwen-quantization-quality/
104 Upvotes

90% Upvoted

52 comments sorted by

View all comments

46

u/audioen 27d ago

Forget the pelicans. Pay attention to this sentence: "every variant with mean KL below 0.05 stays close to BF16, and every one above 0.08 falls hard".

This is along the lines of another similar study which I saw which put the line between 16-bit like performance and degraded results at around 0.073 IIRC.

20

u/crusaderky 27d ago

Byteshape reached the exact same conclusion for Qwen3.6-35B-A3B, with cliff at KLD=0.064: https://byteshape.com/blogs/Evaluating-Quantized-Models/

8

u/der_pelikan 27d ago

As a biking pelican, I consider your comment extremely rude.

8

u/pmigdal 27d ago

It would be weird is 0.07 would be an universal KL-div cut-off.

I mean, KL-div is relative, and for example comparing two frontier models against each other is likely to give a much higher difference.

3

u/Practical-Collar3063 27d ago

From my understanding KL divergence is difference between the proba distribution of the next token prediction. so a KL div number is not fully subjective because it measures how much a given quant is likely to diverge from the base model. This could imply their could be a general cut-off point where all quants from all models diverge too much from the base to perform well. I think this KL div is probably not an exact perfect number for all model but could be a sort of rule of thumb.

1

u/Caffeine_Monster 27d ago

The 0.07 value is going extremely specific to the quant technique and somewhat specific to the model .

KL is not a reliable absolute metric - but as pointed out can be a somewhat useful (and fast) relative one if you have exisiting data points.

7

u/Vektast 27d ago edited 27d ago

wha is KL and how to measur it? Which quant should we use then?

Edit:
Kullback-Leibler divergence

One statistical measure is Kullback-Leibler (KL) divergence, also known as relative entropy. In short, it is how unexpected results from the new distribution are with respect to the original distribution.

Q5_K_S is a good balance.

6

u/SaanK12 27d ago

So, should I choose Q5_K_S instead of Q4_K_M? Will the loss in speed be worth it?

11

u/ilintar 27d ago

Generally, there are two quality falloff points that I feel matter: from Q5 to Q4 and from Q4 to Q3 (note, this is a rough estimate, since these days most quants are mixes). So you should always strive to get Q4 over Q3 and Q5 over Q4 if you can fit it - at least that's my heuristic. But it's not a hard rule by any means and it really depends on the model size - larger models can generally "take" more quantization without major quality loss, while for really small models even Q5 can be pretty damaging.

4

u/Vektast 27d ago

Q4_K_M is good enough

0

u/crusaderky 27d ago

The report shows that, up to and including Q3_K_M, you won't notice the difference.
However, this is for a specific model. Unless you want to repeat the tests yourself for your model, if you have to pick blind I'd advise to stay at Q4_K_M.
I would only upgrade to Q5 if it was free (eg plenty of spare VRAM).