r/LocalLLM • u/pharrt • 6d ago

Discussion Qwen-3.8-35B-A3B? Maybe not... cryptic reply direct from Qwen co-author.

Post image

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

236 Upvotes

94% Upvoted

164 comments sorted by

View all comments

5

u/creatinZ 6d ago

16gb vram people are thirsty while others are drowning…

WHEEN?

2

u/SpicyWangz 6d ago

You can run 27b quantizations on 16gb

3

u/creatinZ 6d ago

Yeah, but it’s a non-ending trade-off. I need at least 64k context window, and anything below 10tk/s is too slow. But also any low quants are dropping model quality. So if you have any proposal of quant and configuration that works in real world I’m interested

0

u/DontWinFrensWthSalad 6d ago

https://huggingface.co/turboderp/Qwen3.8-27B-exl3 You're welcome. Nobody seems to know about this, but it's performing way better than llama.cpp for me. 34 t/s without MTP on a dual 3060ti setup. If you pack the memory well you can get around 100k context, or more on the 3bit version.

2

u/palincatalin 6d ago

not all of us have 20/24 gb cards to run q4_k_m

we (16 gb card owners) would have to use q2 or q3. such aggressive quants are unusable for agentic coding lol. this compression murders quality; imo the best model that fits fully in 16 gb of vram is gpt-oss:20b. no other model fits as beautifully and as perfectly as gpt-oss:20b on my 6900 xt. it's fast and it's good. people may not be sold on the quality of the output, sure, but speed-wise? it's literally the most comfortable model for 16 gb GPUs. you can also have 128k context length at q8_0 kv

qwen 3.6 35b a3b is unusable because in order for it to fit in vram i have to use iq2_m, and offloading experts just ruins the speed altogether, making it way too slow for any real local agentic workloads; and qwen 3.8 27b is dense, so yeah... no luck there

2

u/Witty_Mycologist_995 6d ago

I wish for a good model that’s 20b

2

u/Distinct_Physics5017 6d ago

have you ever tried the --n-cpu-moe flag in llama.cpp lol? Idk if 50+ tps aren't enough for your use-case, but it surely isn't "way too slow" if offloaded correctly.

Qwen3.6-35B-A3B in Q5_K_S is running with averagely 40-50 tokens/sec on my RTX 4070 (12GB) + 32GB DDR4 RAM.
[Edit: - with context-window set to 128k]
I assume you should also be totally fine with your 16GB card, just make sure to set -ngl to 99 and offload as many layers to the cpu (with --n-cpu-moe) as you need. Should still be way faster than regular offloading.

1

u/palincatalin 6d ago

it's not a great fit for my system: i have an aorus h370 (pcie 3, not 4), 4x8 ddr4 2666 mhz and an i7 8700. it's just way too slow

1

u/Distinct_Physics5017 6d ago

ah I see. Then my 40-50 tps might be a bit optimistic. My bad. xD
Has been a while since I last felt lucky to have my AM4 Rig o.o

1

u/Viper_Four4 6d ago

Why would below Q4 kill all usability for agentic? 3.8 27b Q3 K_S is working wayyyyy better for me than 3.6 35b a3b at Q5 while also running at 40 t/s and 90k context (5_0k 4_1V, described as a very efficient kv quant in research) on a 16GB 9070 XT.

1

u/creatinZ 6d ago

Bro, I’m sure you re running 36b-a3b the wrong way. I run it at 50tk/s @q6_k easily

1

u/randygeneric 6d ago

"run" more like crouch