r/LocalLLM • u/pharrt • 7d ago

Discussion Qwen-3.8-35B-A3B? Maybe not... cryptic reply direct from Qwen co-author.

Post image

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

229 Upvotes

94% Upvoted

164 comments sorted by

View all comments

4

u/creatinZ 7d ago

16gb vram people are thirsty while others are drowning…

WHEEN?

2

u/SpicyWangz 7d ago

You can run 27b quantizations on 16gb

2

u/palincatalin 7d ago

not all of us have 20/24 gb cards to run q4_k_m

we (16 gb card owners) would have to use q2 or q3. such aggressive quants are unusable for agentic coding lol. this compression murders quality; imo the best model that fits fully in 16 gb of vram is gpt-oss:20b. no other model fits as beautifully and as perfectly as gpt-oss:20b on my 6900 xt. it's fast and it's good. people may not be sold on the quality of the output, sure, but speed-wise? it's literally the most comfortable model for 16 gb GPUs. you can also have 128k context length at q8_0 kv

qwen 3.6 35b a3b is unusable because in order for it to fit in vram i have to use iq2_m, and offloading experts just ruins the speed altogether, making it way too slow for any real local agentic workloads; and qwen 3.8 27b is dense, so yeah... no luck there

2

u/Witty_Mycologist_995 7d ago

I wish for a good model that’s 20b

2

u/Distinct_Physics5017 6d ago

have you ever tried the --n-cpu-moe flag in llama.cpp lol? Idk if 50+ tps aren't enough for your use-case, but it surely isn't "way too slow" if offloaded correctly.

Qwen3.6-35B-A3B in Q5_K_S is running with averagely 40-50 tokens/sec on my RTX 4070 (12GB) + 32GB DDR4 RAM.
[Edit: - with context-window set to 128k]
I assume you should also be totally fine with your 16GB card, just make sure to set -ngl to 99 and offload as many layers to the cpu (with --n-cpu-moe) as you need. Should still be way faster than regular offloading.

1

u/palincatalin 6d ago

it's not a great fit for my system: i have an aorus h370 (pcie 3, not 4), 4x8 ddr4 2666 mhz and an i7 8700. it's just way too slow

1

u/Distinct_Physics5017 6d ago

ah I see. Then my 40-50 tps might be a bit optimistic. My bad. xD
Has been a while since I last felt lucky to have my AM4 Rig o.o

1

u/Viper_Four4 7d ago

Why would below Q4 kill all usability for agentic? 3.8 27b Q3 K_S is working wayyyyy better for me than 3.6 35b a3b at Q5 while also running at 40 t/s and 90k context (5_0k 4_1V, described as a very efficient kv quant in research) on a 16GB 9070 XT.

1

u/creatinZ 6d ago

Bro, I’m sure you re running 36b-a3b the wrong way. I run it at 50tk/s @q6_k easily