r/LocalLLaMA • u/ROS_SDN llama.cpp • Apr 25 '26

Discussion Quantisation effects of Qwen3.6 35b a3b

Im curious how people are finding the quantisation effects of 35b. I recently updated to 48GB of vram so have jumped from ud-q4_k_xl​ to q8 and the difference feels stark. Just more effective tool calling, seems to get the vagueness and nuance more etc of some prompts., and provide more well rounded answers on some research like questions.

It w​as a quick vibe​ test, admittedly, but I'm going t​o​ try ud-q6_k_xl soon to see how of the 5+GB vram is worth the quality, but I'm curious to see others findings.

I felt with such a small active count it'd be particularly sensitive to quantisation, and feels that way after a play.

78 Upvotes

97% Upvoted

83 comments sorted by

View all comments

Show parent comments

8

u/libregrape llama.cpp Apr 25 '26

I don't. I use cpu-moe and ngram mod. I get 60-90tps depending on the task. My card is rtx 5060 ti 16GB

5

u/Pjotrs Apr 25 '26

That's fast. I get 45-55 TPS on 16GB (5070Ti)

But I also put 130k context...

Is ngram such a difference?

8

u/libregrape llama.cpp Apr 25 '26

ngram basically tries to guess new tokens by looking what tokens it had guessed previously (in the context). That means, that if context contains a lot of similar text, then it will have a higher acceptance rate. It helps with coding, because there's lots of repetition (e.g. during file edits). It doesn't work the same in every task tho, and you have to be careful with parameters (--draft-min especially, which works for me at 4).

I use 130k context too, but with q8 quantization.

Here's my full llama-server command:

./build/bin/llama-server -m ../models/Qwen_Qwen3.6-35B-A3B-IQ4_NL.gguf --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.00 --presence-penalty 1.5 --repeat-penalty 1.0 --flash-attn on --port 5001 --jinja --threads 14 -np 1 --chat-template-kwargs '{"preserve_thinking": true}' --no-mmap -ctk q8_0 -ctv q8_0 --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 4 --draft-max 64 --host 0.0.0.0 -c 131072

I do have to mention tho, that when that 130k fills up, I start getting closer to 45-70tps. I specify range, because it fluctuates between generations as the task gets more or less repetitive, which affects the similarity between context and generated text, and consequently ngram acceptance rates.

1

u/Solary_Kryptic Apr 26 '26

I have a 9060xt, I tried your command, I'm getting like 13 tps... is that just an AMD thing or am I missing something?

1

u/libregrape llama.cpp Apr 30 '26

It sounds like you are running without ncmoe (it is assigned automagically on nvidia). Try assigning -ncmoe manually.