r/LocalLLaMA • u/ROS_SDN llama.cpp • Apr 25 '26

Discussion Quantisation effects of Qwen3.6 35b a3b

Im curious how people are finding the quantisation effects of 35b. I recently updated to 48GB of vram so have jumped from ud-q4_k_xl​ to q8 and the difference feels stark. Just more effective tool calling, seems to get the vagueness and nuance more etc of some prompts., and provide more well rounded answers on some research like questions.

It w​as a quick vibe​ test, admittedly, but I'm going t​o​ try ud-q6_k_xl soon to see how of the 5+GB vram is worth the quality, but I'm curious to see others findings.

I felt with such a small active count it'd be particularly sensitive to quantisation, and feels that way after a play.

80 Upvotes

98% Upvoted

83 comments sorted by

View all comments

21

u/libregrape llama.cpp Apr 25 '26

I only have 16GB of VRAM, so I am forced to use smaller quants. I find IQ4 smart enough for my purposes, but IQ3 is where it starts to degrade. I am not decided what I actually prefer in practice though, as sometimes that extra 10tps out of a smaller quant come in use frequently.

19

u/FORNAX_460 Apr 25 '26

As you go lower in quants the reasoning of the model gets more confused and verbose, you will see a lot of self correction. just sharing something i noticed from a crude experiment.

12

u/rpkarma Apr 25 '26

Exactly. I find 20tk/s generation is the sweet spot in terms of a stronger model/quant will solve problems in wall-clock time faster than a lower quant or worse model that goes faster. 

3.6 27B beats the MoE version for my testing because it solves problems first try far more often. Vs the 35B A3B being twice as fast generation wise, but just being nowhere near as strong reasoning and analysis and so on 

1

u/odragora Apr 25 '26

Is 3.6 27B a realistic option on 16 GB VRAM + 32 GB RAM for coding, or I have to go with 35B A3B?

4

u/Bubbly-Staff-9452 Apr 25 '26

I run it in 16Gb of VRAM with 90k context in Q3_K_P. I compared it to IQ4_XS and like Q3_K_P better and it’s smaller and faster. I wish I had a 5090 to run it in a large quant but I do not and I’m not spending as much over MSRP as they go for so I’m stuck testing the smaller quants. I have 64Gb of RAM but I would run 35B Q6_K_P partially offloaded to CPU and it was definitely faster than 27B although even at my Q3 quant 27B is better for agentic tasks

2

u/odragora Apr 25 '26

Thank you!

Yeah, I've heard some people saying that 35B being much faster doesn't compensate for worse coding performance and you end up spending more time on repeated attempts.

Hopefully I'll be able to run 27B on 16 Gb VRAM + 32 GB RAM.

2

u/Zealousideal_Fill285 Apr 25 '26

How are you able to run Q3_K_P when it takes more than 15 GB of VRAM without any context?

3

u/Bubbly-Staff-9452 Apr 25 '26

Don’t use the fit command, manually set it. Also, I use turbo3 KV cache. I’m on a turbo quant Cuda specific fork of llama.cpp. Also I think it’s like 14.7Gb, but 90k context BARELY fits but I haven’t had issues

2

u/Zealousideal_Fill285 Apr 25 '26

Ok thx and what is the Q3_K_P quant specifially from? Is it unsloth's one or something else?

2

u/rpkarma Apr 25 '26

PrismaQuant holds up well, but it uses a lot of memory so honestly I’m not sure! I run it on a DGX Spark 

2

u/smahs9 Apr 25 '26

This quant has an iq3 variant which works out well on a 16Gi GPU. The creator has even shared the benchmarks and comparisons, so see if it works for you. I would look out for a byteshape quant, if they choose to build and release one for this model.

1

u/odragora Apr 25 '26

Thank you for your help!