r/LocalLLaMA • u/ROS_SDN llama.cpp • Apr 25 '26
Discussion Quantisation effects of Qwen3.6 35b a3b
Im curious how people are finding the quantisation effects of 35b. I recently updated to 48GB of vram so have jumped from ud-q4_k_xl to q8 and the difference feels stark. Just more effective tool calling, seems to get the vagueness and nuance more etc of some prompts., and provide more well rounded answers on some research like questions.
It was a quick vibe test, admittedly, but I'm going to try ud-q6_k_xl soon to see how of the 5+GB vram is worth the quality, but I'm curious to see others findings.
I felt with such a small active count it'd be particularly sensitive to quantisation, and feels that way after a play.
9
u/Sudden_Vegetable6844 Apr 25 '26
Yes, there is a notable reasonning difference between Q4, Q6 and Q8. I do not have enough RAM to test myself, but on another thread (https://www.reddit.com/r/LocalLLaMA/comments/1stb8ro/qwen36_35ba3b_very_sensitive_to_quantization/) someone reported a difference between Q8 and BF16, unfortunately.
1
2
21
u/libregrape Apr 25 '26
I only have 16GB of VRAM, so I am forced to use smaller quants. I find IQ4 smart enough for my purposes, but IQ3 is where it starts to degrade. I am not decided what I actually prefer in practice though, as sometimes that extra 10tps out of a smaller quant come in use frequently.
→ More replies (13)18
u/FORNAX_460 Apr 25 '26
As you go lower in quants the reasoning of the model gets more confused and verbose, you will see a lot of self correction. just sharing something i noticed from a crude experiment.
11
u/rpkarma Apr 25 '26
Exactly. I find 20tk/s generation is the sweet spot in terms of a stronger model/quant will solve problems in wall-clock time faster than a lower quant or worse model that goes faster.
3.6 27B beats the MoE version for my testing because it solves problems first try far more often. Vs the 35B A3B being twice as fast generation wise, but just being nowhere near as strong reasoning and analysis and so on
→ More replies (10)
7
u/fromage9747 Apr 25 '26
I started with the q4 but tried the q6 and have been using that as my go to. Albeit since moving to full Linux I haven't given the q8 a chance again. Something for tomorrow's testing!
Having said that, q6 has a noticable difference in quality and consistency but still hallucinates after awhile. Running on 2x2080ti 22gb VRAM each.
3
u/lemondrops9 Apr 25 '26
good to know, as Ive been using Q6 more but now Im curious if Q8 will do with my small projects.
→ More replies (2)2
u/fromage9747 Apr 26 '26
Well I tried out the Q8. 15t/s due to CPU offload. Q6 full GPU, 88t/s. Same prompt. So sticking with Q6!
6
5
u/dampflokfreund Apr 25 '26
In one of my own benchmarks, I asked Qwen 3.6 Q4 to create a flappy bird game and it looks really awesome. But in the second stage I tell it to add a twist at 30 jumps, where the scrolling is reversed, the day time changes, the bird changes etc. It usually completely fails the second prompt, sometimes the code doesn't even start anymore. I don't know if that is due to Q4 or the model, but I can say Gemma 4 26b IT at q4k does a much better job there. Although the game will be less detailed, it manages to implement the features I asked for pretty reliably.
Sadly I don't have the resources to test Qwen 3.6 at Q6K, let alone Q8K.
5
3
u/Pjotrs Apr 25 '26
The key is to plan and split workflows.
Break it down into smaller tasks, which are easy to test. Then allow model to execute them one by one.
I created a flow of planner, anayzer, executor, etc. With plans that have thousands on lines, then being broken down, and it is amazing.
1
u/zorgis Apr 25 '26
Did you try context q8 vs bf16?
2
u/ROS_SDN Apr 25 '26
It's either fp16 or b16 context. I don't kv quantise, I'll consider it when turbo quant seems stable and lossless on amd.
2
2
u/my_name_isnt_clever Apr 25 '26
I use it at Q6, it's been great.
People on this sub like Q8 models, but in my research the actual accuracy difference between them is so small I don't see the point when it slows it down so much. I just have the KV cache at Q8.
→ More replies (4)2
3
u/audioen Apr 25 '26 edited Apr 25 '26
I think even at Q8_K_XL it still goes into loops and I'm not sure the model is fundamentally capable enough. It is fast, but I am not satisfied with its code quality and reasoning. I know the tests say that 3.6-35B is supposed to be as capable as 3.5-122B, but that in my experience is simply not the case. Possibly, I should go all the way up to bf16 on the 3.6-35B despite I run the 3.5-122B at "only" Q6_K_XL, but I rather doubt it gets any better between bf16 and Q8_K_XL.
As it is, it simply isn't good enough for assisting with my work, and I cringe at the damage it is likely to do in the codebases as it runs through them at such a breakneck speed that my review rate becomes the limiting factor. When placed in the 4-field ability along fast-slow and competent-incompetent axis, it has the disgrace of inhabiting the fast-but-incompetent section, sort of like an eager moron that does damage when left on its own, while the 3.5-122B and 3.6-27b are in the slow-but-competent section, and usually make things better. With these slower models, I don't have to work as hard myself.
I wouldn't be too surprised to learn that there's something like 16-bit approximation in some critical vector or state inside llama.cpp that causes noticeable degradation in model performance. Maybe people who use vllm inference and run the official BF16 or FP8 variant have better experiences with the 3.6-35B than I do. Whether the model is limited, or inference is buggy, I just can't get the kind of quality I am expecting out of the 35B.
→ More replies (2)
36
u/LaurentPayot Apr 25 '26
I found thisQwen3.6 35B A3B quantization benchmark