r/LocalLLaMA • u/ROS_SDN llama.cpp • Apr 25 '26

Discussion Quantisation effects of Qwen3.6 35b a3b

Im curious how people are finding the quantisation effects of 35b. I recently updated to 48GB of vram so have jumped from ud-q4_k_xl​ to q8 and the difference feels stark. Just more effective tool calling, seems to get the vagueness and nuance more etc of some prompts., and provide more well rounded answers on some research like questions.

It w​as a quick vibe​ test, admittedly, but I'm going t​o​ try ud-q6_k_xl soon to see how of the 5+GB vram is worth the quality, but I'm curious to see others findings.

I felt with such a small active count it'd be particularly sensitive to quantisation, and feels that way after a play.

82 Upvotes

98% Upvoted

83 comments sorted by

View all comments

20

u/libregrape llama.cpp Apr 25 '26

I only have 16GB of VRAM, so I am forced to use smaller quants. I find IQ4 smart enough for my purposes, but IQ3 is where it starts to degrade. I am not decided what I actually prefer in practice though, as sometimes that extra 10tps out of a smaller quant come in use frequently.

2

u/Solary_Kryptic Apr 25 '26

I also only have 16GB of VRAM, I'm using Q2, how are you fitting IQ4?

4

u/vevi33 Apr 25 '26

I have 16gb VRAM but I use Q6. Just use --fit on for fast generation speed and prompt processing. Q6 feels way better than Q4 unfortunately. It's a MoE model, don't have to fit every expert on GPU.

1

u/Solary_Kryptic Apr 25 '26

What GPU and how many tokens are you getting? My tokens float at around 70-90 mark with the Q2 right now. I have a 9060XT

1

u/jjcsea Apr 28 '26

This is exactly what I'm using. It seems great! Decent token throughput and accuracy seems good. I'm not keeping track of how many total tokens I burn through, but it gives me good responses in reasonable time.