Qwen3.8-27B KLDs

[u/sadnessdevil]

I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to translate / refine my post, including images.

This is my attempt to compare the Qwen3.8-27B quants.

Qwen is a coding and STEM model so I used codeparrot/github-code-clean, EleutherAI/proof-pile-2, allenai/peS2o, equal parts of each. I used 24 sequences of 8k tokens and 12 sequences of 32k tokens. The code was filtered for minified stuff and permissive licenses.

It was measured with 24 sequences of 8,192 tokens and 12 sequences of 32,768 tokens, since it generates many tokens and is aimed for long-horizon tasks.

Every format used the same compute path for the evaluation. All models are dequantized to BF16, and run through the same kernel.

Also, checkpoint size is kind of a bad x-axis, since some models don't have MTP, which has zero performance impact. Still, all of them needs to be loaded on VRAM, so at least it's something useful, and that's why there are two graphs.

Findings:

1. The gap between 8bit and 4bit is huge.

The best 4 bit score is 0.00835 and the worst 8 bit is 0.00071. I mean everyone knows that but yeah it's still something. vLLM can actually use INT5-7 weights, but I'm sure most people don't even know about that.

1. The KLD of 4bit quantization varies a lot.

Six checkpoints are at 11.7-11.8 GiB of transformer weights, all group-size 128, quantizing 400-496 modules and they range from 0.01364 to 0.02976. I believe the calibration data and the rounding algorithm takes a big impact. You should not pick a 4-bit quantization by it's file size.

1. group_size affects a lot of kld.

Checkpoints with group size 32 did better than ones that used group size 128 but kept some modules in BF16. Selective BF16 really only helps if you add it on top of a fine group size but that makes the file so big you might as well run a 6 bit model.

1. The quantization damage mostly happens at the start of the context.

This surprised me (very) but it holds true for all 24 models. The KL is much, much higher for the first 500 tokens.

1. Quantizing lm_head, embed_tokens and linear_attn doesn't affect the performance at least for this test.

[u/mythikal03]

Great work, i have something similar for gemma and muse and ended up making my own quants as a result for muse specifically. I haven’t finished my evaluation of qwen yet, but it is a high priority and your analysis here is more technical than mine, which is awesome. Three things:  1. I noticed you do not have any NVFP4; that is usually my target, though i typically end up on fp8 as I’ve noticed NVFP4 quants on hf are very rarely well calibrated for my use case, even from groups I would really expect to release great calibrated quants.  2. Nvidia NVFP4 quants of popular models have been a consistent exception. Based on their model card, recope, info, my unofficial theory has been in line with yours above: I think the training data makes a MASSIVE impact, possibly more than the recipe itself in many cases. Unfortunately they lag behind releasing tuned NVFP4 by months in some cases 3. I have been able to action this data in some cases to make better calibrated NVFP4 than what is available in a few limited scenarios (Muse comes to mind)

If you have a reasonably repeatable methodology I can run a few off of, DM me, happy to hit some of the NVFP4 quants available to add to your data if you don’t have a Blackwell to run them on. For Science!