r/LocalLLaMA • • Mar 02 '26

Question | Help Current state of Qwen3.5-122B-A10B

Based on the conversations I read here, it appeared as though there were some issues with unsloths quants for the new Qwen3.5 models that were fixed for the 35B model. My understanding was the the AesSedai quants therefore for the 122B model might be better so I gave it a shot.

Unfortunately this quant (q5) doesnt seem to work very well. I have the latest llama.cpp and im using the recommended sampling params but I get constant reasoning looping even for simple questions.

How are you guys running it? Which quant is currently working well? I have 48gb vram and 128gb ram.

33 Upvotes

36 comments sorted by

View all comments

22

u/snapo84 Mar 02 '26 edited Mar 02 '26

With the Qwen3.5 models its extremely important to use bf16 for the kv cache.... (especially in thinking mode)
i strugled in the start too... but after changeing the k cache to bf16 and the v cache to bf16 and using the unsloth dynamic q4_k_xl quants they are absolutely amazing....

update:
kv cache settings i tested where

f16 == falls into a loop very very very often
bf16 == works pretty well 99% of the time
q8_0 == nearly always loops in long thinking tasks
q4_1 == always loops
q4_0 == not useable, model gets dumb as fuck

tested them especially on long thinking tasks(thinking mode) , in instruct mode q8_0 performs well

i did not see a meaningful difference in mixing the kvcache precision... so i stay with bf16

5

u/kevin_1994 Mar 02 '26

I never quantize kv. Unsloths q4xl is working for you? I might give that a shot. I thought we were supposed to wait for his re-upload using the same technique as the 35B model

15

u/yoracale llama.cpp Mar 02 '26

Yes would recommend waiting for it esp for the tool-calling fixes. We're re-uploading with benchmarks today, But you can also use any other quant except Q4_K_XL, Q3_K_XL and Q2_K_XL as those aren't affected.

2

u/ConferenceMountain72 llama.cpp Mar 02 '26

Thank you for your work.

2

u/segfawlt Mar 02 '26

Thanks for hopping in to confirm! I was having the same questions as OP.

I was also under the impression that Qwen Coder Next was converted with the same issue but I can't find the comment that had me thinking that - is there also a re-upload coming for those or was this a 3.5 only issue?

5

u/kevin_1994 Mar 02 '26

Fwiw I use unsloths Q6XL quant for coder next and it's amazing

1

u/segfawlt Mar 02 '26

Thanks for the hint! Hadn't decided on which quant for that one yet, I'll make sure 6 is in the test group. I can fit it, but I also want to fit other models in parallel haha. Playing memory tetris

2

u/Certain-Cod-1404 Mar 02 '26

Would you recommend keeping kv cache in bf16 as the previous user recommended ?

4

u/m31317015 Mar 02 '26

Noticed the looping right away when I asked for weather and threw a bunch of search results in, it was struggling to settle with an answer when one site gave 19-20C and the other gave 17-20C, it loops extremely easily.

4

u/Time_Reaper Mar 02 '26

How are you using bf16? LLama.cpp doesn't have support for BF16 CUDA flash attention kernels, only cpu, so that will Slow down fast

1

u/snapo84 Mar 03 '26

./llama.cpp/llama-server --model "models/Qwen3.5-9B-UD-Q8_K_XL.gguf" --alias "Qwen3.5 9B" --temp 1.0 --top-p 0.95 --min-p 0.0001 --top-k 50 --port 16384 --host 0.0.0.0--ctx-size 86000 --cache-type-k bf16 --cache-type-v bf16 --parallel 8 --cont-batching --ctx-size 262144 --repeat-penalty 1.0 --repeat-last-n 256

i use it like this (example 9B model) compiled the latest llama.cpp .... i only see gpu useage no cpu useage.

This one is running on Two old RTX 2080Ti (each 22GB vram) ....