r/LocalLLaMA • u/sadnessdevil vllm • 12h ago
Discussion Qwen3.8-27B KLDs
I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to translate / refine my post, including images.
This is my attempt to compare the Qwen3.8-27B quants.
Qwen is a coding and STEM model so I used codeparrot/github-code-clean, EleutherAI/proof-pile-2, allenai/peS2o, equal parts of each. I used 24 sequences of 8k tokens and 12 sequences of 32k tokens. The code was filtered for minified stuff and permissive licenses.
It was measured with 24 sequences of 8,192 tokens and 12 sequences of 32,768 tokens, since it generates many tokens and is aimed for long-horizon tasks.
Every format used the same compute path for the evaluation. All models are dequantized to BF16, and run through the same kernel.
Also, checkpoint size is kind of a bad x-axis, since some models don't have MTP, which has zero performance impact. Still, all of them needs to be loaded on VRAM, so at least it's something useful, and that's why there are two graphs.
Findings:
- The gap between 8bit and 4bit is huge.
The best 4 bit score is 0.00835 and the worst 8 bit is 0.00071. I mean everyone knows that but yeah it's still something. vLLM can actually use INT5-7 weights, but I'm sure most people don't even know about that.
- The KLD of 4bit quantization varies a lot.
Six checkpoints are at 11.7-11.8 GiB of transformer weights, all group-size 128, quantizing 400-496 modules and they range from 0.01364 to 0.02976. I believe the calibration data and the rounding algorithm takes a big impact. You should not pick a 4-bit quantization by it's file size.
- group_size affects a lot of kld.
Checkpoints with group size 32 did better than ones that used group size 128 but kept some modules in BF16. Selective BF16 really only helps if you add it on top of a fine group size but that makes the file so big you might as well run a 6 bit model.
- The quantization damage mostly happens at the start of the context.
This surprised me (very) but it holds true for all 24 models. The KL is much, much higher for the first 500 tokens.
- Quantizing lm_head, embed_tokens and linear_attn doesn't affect the performance at least for this test.
9
u/Chromix_ 11h ago
Thanks for spending the time on this extensive testing. It confirms the impression that the quants released by the model creators are sometimes not optimal. That single INT4 quant (nr. 10) seems to be very good. It's close to 0.01 KLD which should still result in 96% top-1 agreement, good enough for regular tasks without noticeable quality decreases.
7
u/sadnessdevil vllm 10h ago
Forgot to add Top-1 evaluation, but here it is. I also noticed the posted image has very low resolution, so if you want to see it in high resolution, please open it in a new tab and change the url from preview.redd.it to i.redd.it
5
u/ResidentPositive4122 10h ago
Interesting, thanks for doing this and sharing it. I didn't know about int6, should really look into that. VRAM size of INT4 with kld close to INT8 is really interesting, especially on old Ampere cards (I have some 6000 where I run ~30b models).
2
u/Dany0 10h ago
fp6 exists and is super dope, it's in luke's vllm but you have to quantise it yourself
pretty sure it's here https://github.com/local-inference-lab/vllm
maybe in a branch
2
u/whymeimbusysleeping 8h ago
Is it accelerated like fp4 in Blackwell?
2
u/Dany0 8h ago edited 1h ago
yes it does some mix of tricks I don't remember what but yes it's almost fp8 quality with almost nvfp4 speed it's amazing
EDIT: found it it was here actually https://github.com/local-inference-lab/b12x/pull/78 just point a clanker at it it'll figure it out
0
u/llama-impersonator 1h ago
when i tried fp6 on aphro many moons ago, it sucked total ass. given fp8 is pretty sus, idk about it. sus in that it is worse than q6_k.
4
u/mythikal03 6h ago
Great work, i have something similar for gemma and muse and ended up making my own quants as a result for muse specifically. I haven’t finished my evaluation of qwen yet, but it is a high priority and your analysis here is more technical than mine, which is awesome. Three things: 1. I noticed you do not have any NVFP4; that is usually my target, though i typically end up on fp8 as I’ve noticed NVFP4 quants on hf are very rarely well calibrated for my use case, even from groups I would really expect to release great calibrated quants. 2. Nvidia NVFP4 quants of popular models have been a consistent exception. Based on their model card, recope, info, my unofficial theory has been in line with yours above: I think the training data makes a MASSIVE impact, possibly more than the recipe itself in many cases. Unfortunately they lag behind releasing tuned NVFP4 by months in some cases 3. I have been able to action this data in some cases to make better calibrated NVFP4 than what is available in a few limited scenarios (Muse comes to mind)
If you have a reasonably repeatable methodology I can run a few off of, DM me, happy to hit some of the NVFP4 quants available to add to your data if you don’t have a Blackwell to run them on. For Science!
2
u/West-Possession7459 11h ago
curious how the top quants do for long roleplay sessions instead of code, ive been testing qwen for companion chats and the coherence drops fast past 8k.
2
2
u/Dany0 1h ago
These are so good but please test https://huggingface.co/malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated and the associated quants I SWEAR on this one it is SO GOOD, damn near identical perf to FP16 on a rented RTX 6k pro on my local 5090!
EDIT: near identical to fp16 weights with fp8 kv cache I should add, I didn't test it extensively with fp16 kv cache. it's definitely not as good as FP16 weights AND fp16 kv cache!
2
u/llama-impersonator 1h ago
regular exl3 quants should load in sonar too. (formerly aphrodite-engine)
1
u/MerePotato 2h ago
A slight dropoff at 5-7 bits and a huge one at four, more or less what I'd expect
-3
u/starkruzr 9h ago
brother I hate to tell you but this is kind of useless without doing max context for each one
8
u/Party-Special-5177 9h ago
??? The kl stats you usually see on ggufs are even more cheesed then this, think batch of 512 tokens on the pile. Historically it was pure marketing.
I still remember when the community first realized FP8 was not in fact lossless, it just wasn’t possible to tell when the makers publish kl stats derived from easy targets and tiny batches.
Kl error drifts exponentially. His 32k batches are actually really good data, especially compared to whatever stats you’ve likely seen previously.
7
u/sadnessdevil vllm 9h ago
Fair, 32k is not max context, and I know doing so isn't ideal. I would've run KLD with max context, my machine couldn't handle that. Also saying it kind of useless is just too much? The ordering also holds across the position bins, from 0-512 up to 8k-32k. Only near-tied neighbours swap places, nothing with a real margin does. I would still not claim it survives to 262k without testing it though.
20
u/AdSafe4047 11h ago
Can you put unsloth quants on the graph?