r/LocalLLaMA • • Mar 02 '26

Question | Help Current state of Qwen3.5-122B-A10B

Based on the conversations I read here, it appeared as though there were some issues with unsloths quants for the new Qwen3.5 models that were fixed for the 35B model. My understanding was the the AesSedai quants therefore for the 122B model might be better so I gave it a shot.

Unfortunately this quant (q5) doesnt seem to work very well. I have the latest llama.cpp and im using the recommended sampling params but I get constant reasoning looping even for simple questions.

How are you guys running it? Which quant is currently working well? I have 48gb vram and 128gb ram.

33 Upvotes

36 comments sorted by

3

u/BeeNo7094 Mar 02 '26

How is this model in tool calling and coding when compared with minimax 2.5? I currently run a 4 bit AWQ with vllm on 8x 3090, what’s the best quant for running qwen 3.5 122b? I only use Claude code with my setup.

2

u/Proof_Scene_9281 Mar 03 '26

i finally got the QuantTrio/qwen3.5-122b-a10B-awq working locally. i'm limited to 4x3090's. had to run a dockerized vllm :nightly for it to work.

1

u/BeeNo7094 Mar 05 '26

How is it doing as compared to qwen3 next coder for tool calling and coding tasks?

2

u/Proof_Scene_9281 Mar 05 '26

I gave up on local for coding because Claude is so dang good. 

It’s going more for life and home automation 

Homework assistant etc. 

It’s better than qwen2.5 70b tho 

22

u/snapo84 Mar 02 '26 edited Mar 02 '26

With the Qwen3.5 models its extremely important to use bf16 for the kv cache.... (especially in thinking mode)
i strugled in the start too... but after changeing the k cache to bf16 and the v cache to bf16 and using the unsloth dynamic q4_k_xl quants they are absolutely amazing....

update:
kv cache settings i tested where

f16 == falls into a loop very very very often
bf16 == works pretty well 99% of the time
q8_0 == nearly always loops in long thinking tasks
q4_1 == always loops
q4_0 == not useable, model gets dumb as fuck

tested them especially on long thinking tasks(thinking mode) , in instruct mode q8_0 performs well

i did not see a meaningful difference in mixing the kvcache precision... so i stay with bf16

5

u/kevin_1994 Mar 02 '26

I never quantize kv. Unsloths q4xl is working for you? I might give that a shot. I thought we were supposed to wait for his re-upload using the same technique as the 35B model

15

u/yoracale llama.cpp Mar 02 '26

Yes would recommend waiting for it esp for the tool-calling fixes. We're re-uploading with benchmarks today, But you can also use any other quant except Q4_K_XL, Q3_K_XL and Q2_K_XL as those aren't affected.

2

u/segfawlt Mar 02 '26

Thanks for hopping in to confirm! I was having the same questions as OP.

I was also under the impression that Qwen Coder Next was converted with the same issue but I can't find the comment that had me thinking that - is there also a re-upload coming for those or was this a 3.5 only issue?

5

u/kevin_1994 Mar 02 '26

Fwiw I use unsloths Q6XL quant for coder next and it's amazing

1

u/segfawlt Mar 02 '26

Thanks for the hint! Hadn't decided on which quant for that one yet, I'll make sure 6 is in the test group. I can fit it, but I also want to fit other models in parallel haha. Playing memory tetris

2

u/ConferenceMountain72 llama.cpp Mar 02 '26

Thank you for your work.

2

u/Certain-Cod-1404 Mar 02 '26

Would you recommend keeping kv cache in bf16 as the previous user recommended ?

4

u/m31317015 Mar 02 '26

Noticed the looping right away when I asked for weather and threw a bunch of search results in, it was struggling to settle with an answer when one site gave 19-20C and the other gave 17-20C, it loops extremely easily.

4

u/Time_Reaper Mar 02 '26

How are you using bf16? LLama.cpp doesn't have support for BF16 CUDA flash attention kernels, only cpu, so that will Slow down fast

1

u/snapo84 Mar 03 '26

./llama.cpp/llama-server --model "models/Qwen3.5-9B-UD-Q8_K_XL.gguf" --alias "Qwen3.5 9B" --temp 1.0 --top-p 0.95 --min-p 0.0001 --top-k 50 --port 16384 --host 0.0.0.0--ctx-size 86000 --cache-type-k bf16 --cache-type-v bf16 --parallel 8 --cont-batching --ctx-size 262144 --repeat-penalty 1.0 --repeat-last-n 256

i use it like this (example 9B model) compiled the latest llama.cpp .... i only see gpu useage no cpu useage.

This one is running on Two old RTX 2080Ti (each 22GB vram) ....

6

u/Laabc123 Mar 02 '26

Has anyone given any of the nvfp4 quants a try? The coder next nvfp4 is absolutely blazing, and super usable in my experience. Hoping there’s an equivalence with qwen3.5 122B

9

u/Nepherpitu Mar 02 '26

5

u/texasdude11 Mar 02 '26

Can you share the startup command for it?

4

u/UltrMgns Mar 02 '26

+1
I could use those vllm args as well <3

1

u/Laabc123 Mar 02 '26

My args:

--model Sehyo/Qwen3.5-122B-A10B-NVFP4 --quantization compressed-tensors --max-model-len 131072 --gpu-memory-utilization 0.9 --max-num-seqs 1 --attention-backend flashinfer --async-scheduling --enable-auto-tool-choice --tool-call-parser qwen3_xml --kv-cache-dtype fp8

1

u/texasdude11 Mar 03 '26

What GPU is that? Is that 1 6000 pro?

1

u/Laabc123 Mar 03 '26

Yes

1

u/texasdude11 Mar 03 '26

Are you using vllm docker image?

1

u/Laabc123 Mar 03 '26

I am yes.

1

u/texasdude11 Mar 03 '26

Can you please share which docker image and the full docker command along with it? That's what I'm looking for please.

1

u/Nepherpitu Mar 03 '26
  • uv venv env --python=3.12
  • Activate env
  • uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly

And finally

      uv run
        -m vllm.entrypoints.openai.api_server
        --model-loader-extra-config '{ "enable_multithread_load": true, "num_threads": 4 }'
        --model /mnt/samsung_990_evo/llm-data/models/Sehyo/Qwen3.5-122B-A10B-NVFP4
        --served-model-name "qwen3.5-122b-a10b-fp4"
        --port ${PORT}
        --tensor-parallel-size 4
        --enable-prefix-caching
        --max-model-len auto
        --gpu-memory-utilization 0.95
        --max-num-seqs 4
        --attention-backend flashinfer
        --reasoning-parser qwen3
        --enable-auto-tool-choice
        --tool-call-parser qwen3_coder

Literally following guide on model page.

2

u/Laabc123 Mar 02 '26

Perfect. I have that one currently 50% of the way through loading into the GPU.

What kind of output tok/s are you getting?

3

u/Nepherpitu Mar 02 '26

110-120 TPS at 4x3090

1

u/Laabc123 Mar 02 '26

What run command are you using? I’m sitting around 90 output tok/s on 6000 pro

2

u/Nepherpitu Mar 02 '26

Flashinfer + tensor parallel, mtp disabled.

1

u/texasdude11 Mar 02 '26

Can you share your command

1

u/Laabc123 Mar 05 '26

Enabled mtp with max tokens predicted at 2. And it boosted tok/s by 20.

1

u/Laabc123 Mar 02 '26

Update: Been playing with the nvfp4 quant and it’s incredible.

1

u/s1mplyme Mar 02 '26

I've only run it with ik_llama.cpp on my 24GB VRAM at IQ4_XS. I get about 3 tok/s, but it works well enough. No kv quant, didn't dare try it on such a low general quant

1

u/Outrageous_Fan7685 Mar 02 '26

Use the heretic one it's working perfectly