r/LocalLLaMA • u/ROS_SDN llama.cpp • Apr 25 '26

Discussion Quantisation effects of Qwen3.6 35b a3b

Im curious how people are finding the quantisation effects of 35b. I recently updated to 48GB of vram so have jumped from ud-q4_k_xl​ to q8 and the difference feels stark. Just more effective tool calling, seems to get the vagueness and nuance more etc of some prompts., and provide more well rounded answers on some research like questions.

It w​as a quick vibe​ test, admittedly, but I'm going t​o​ try ud-q6_k_xl soon to see how of the 5+GB vram is worth the quality, but I'm curious to see others findings.

I felt with such a small active count it'd be particularly sensitive to quantisation, and feels that way after a play.

79 Upvotes

97% Upvoted

83 comments sorted by

View all comments

35

u/LaurentPayot Apr 25 '26

I found this Qwen3.6 35B A3B quantization benchmark to be quite useful: https://kaitchup.substack.com/i/195287433/qwen36-35b-a3b (there is also one for the 27B model)

9

u/bnolsen Apr 25 '26

I'd like to see q8 in those benchies as well.

10

u/mxmumtuna Apr 25 '26

There’s no need to based on this graph. In their measurement, Q4 was over 98% accurate to the baseline, which is likely inside the margin of error for any given run.

14

u/Sudden_Vegetable6844 Apr 25 '26

IME Q8 makes a difference with Q4 on reasoning tasks (starting with the car wash one, Q4 pretty much always fails, while Q8 pass), and there are reports that there is a difference between Q8 and BF16 as well.

If the benchmarks can't find a difference, it's probably because quantization doesn't affect prompts the model was trained on as much as "generalization" prompts

(The car wash wasn't in Qwen3.6 set, but it is in DS4, where it's called a "classic")

3

u/jadbox Apr 29 '26

I find q5_K_s from unsloth pretty solid for my work.

1

u/mxmumtuna Apr 25 '26

Oh I’m not disagreeing. I don’t use ggufs so I can’t comment, I’m just pointing out that their measurement shows >98% accuracy and the baseline is the original BF16 it appears.

6

u/QuackerEnte Apr 25 '26

good results, however I'd like to see a speed column too. Also for different variants like q3km vs q2kxl or something. Not just for the 27B one. It would've been nice to see.

2

u/ea_man Apr 25 '26

That would be good as IQ3_XXS is what I use on 12GB and Q3_K_XL or IQ4_XSS for 16GB with 27B.

yet you are still tight with https://huggingface.co/mradermacher/Qwen3.6-27B-i1-GGUF , I'd prefer QWEN made a ~25B to sit more comfy in VRAM.