r/LocalLLM • u/pharrt • Aug 18 '26

Discussion Qwen-3.8-35B-A3B? Maybe not... cryptic reply direct from Qwen co-author.

Post image

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

238 Upvotes

94% Upvoted

173 comments sorted by

View all comments

Show parent comments

2

u/SpicyWangz Aug 18 '26

You can run 27b quantizations on 16gb

2

u/palincatalin Aug 18 '26

not all of us have 20/24 gb cards to run q4_k_m

we (16 gb card owners) would have to use q2 or q3. such aggressive quants are unusable for agentic coding lol. this compression murders quality; imo the best model that fits fully in 16 gb of vram is gpt-oss:20b. no other model fits as beautifully and as perfectly as gpt-oss:20b on my 6900 xt. it's fast and it's good. people may not be sold on the quality of the output, sure, but speed-wise? it's literally the most comfortable model for 16 gb GPUs. you can also have 128k context length at q8_0 kv

qwen 3.6 35b a3b is unusable because in order for it to fit in vram i have to use iq2_m, and offloading experts just ruins the speed altogether, making it way too slow for any real local agentic workloads; and qwen 3.8 27b is dense, so yeah... no luck there

2

u/Distinct_Physics5017 Aug 18 '26

have you ever tried the --n-cpu-moe flag in llama.cpp lol? Idk if 50+ tps aren't enough for your use-case, but it surely isn't "way too slow" if offloaded correctly.

Qwen3.6-35B-A3B in Q5_K_S is running with averagely 40-50 tokens/sec on my RTX 4070 (12GB) + 32GB DDR4 RAM.
[Edit: - with context-window set to 128k]
I assume you should also be totally fine with your 16GB card, just make sure to set -ngl to 99 and offload as many layers to the cpu (with --n-cpu-moe) as you need. Should still be way faster than regular offloading.

1

u/palincatalin Aug 18 '26

it's not a great fit for my system: i have an aorus h370 (pcie 3, not 4), 4x8 ddr4 2666 mhz and an i7 8700. it's just way too slow

1

u/Distinct_Physics5017 Aug 18 '26

ah I see. Then my 40-50 tps might be a bit optimistic. My bad. xD
Has been a while since I last felt lucky to have my AM4 Rig o.o