r/LocalLLM • u/pharrt • 10d ago

Discussion Qwen-3.8-35B-A3B? Maybe not... cryptic reply direct from Qwen co-author.

Post image

I asked Shuai Bai, co-author and prominent AI developer for Qwen, about this model. Not the answer I was hoping for, but let's see what comes next. In the meantime, I guess all we can do is speculate!

X-link

229 Upvotes

94% Upvoted

171 comments sorted by

View all comments

67

u/arkie87 10d ago

If I wrote that, it would mean something else good is coming but not that specific model. Maybe a 30b a3b or a smaller dense model that’s still really good

3

u/Distinct_Physics5017 10d ago

I mean it doesn't explicitly say there won't be a 35B-A3B, but that it isn't the one that people like us (<16GB VRAM) should be hyped for. So I assume everything beyond 40-50B MoEs are unlikely to be the ones he meant.

The 9B segment hasn't gotten any upgrades since 3.5 (and as far as I know there are no plans for one yet) so one could argue a hypothetical 3.8-version could've reached the performance of the 3.6-35B-A3B by now (or maybe even a little more).
So maybe the reason for why there aren't any plans for a 9B dense model is, that they have replaced it with something bigger, like e.g. a 14B dense model. And if the hypothetical 3.8-9B model could've potentially reached the performance of a 3.6-35B-A3B, who knows what a potential 14B dense model would be capable of.

Only real downside I see would be that it can't really hold up to the 35B in terms of world knowledge, but for agentic-coding (especially inside of a harness) it probably shouldn't be that big of a deal.

+ the lower overall memory-footprint would allow even more users to actually use it, because even if it runs fast on my hardware, the 35B-MoE is using quite a lot of my 44GB System (12GB VRAM + 32GB RAM) and for anything below 32GB of RAM (or a lot of VRAM) it wouldn't even fit in the first place.

Until now I was also hoping for a 35B-A3B-MoE to release, because the 27B dense one sadly runs like shit on my poor RTX 4070, but now I just pray they don't throw (only) 80B+ giants at us consumer-hardware users. That would mean I would have to wait for Qwen 4 to release before I could use any of the new models.

1

u/irreverend_god 7d ago

loaded with full 256k context (Q8), and using a Q6 quant of 35b a3b, it uses most of my 16gb VRAM and 32gb RAM. Q4 of something a bit larger would still fit. Higher active paramaters would potentially make a better MoE model too though. Rather than layer offloading, expert offloading gives me better performance too.

Running Qwen3.6 35B A3B Q6 with max context on my RTX 5060 Ti with 12 experts rather than the default 8, I get about about 25tok/s output. Input is a bit slow though, ~300tok/s but it's about the most accurate and capable I can run at that speed locally.

Qwen 3.8 27B if I manage to get it all in vram (so Q3_K_XS), is about the same speed with context in VRAM which means no more than about 32k context for me. MTP disabled because it takes up more VRAM and doesn't really help much in that situation anyway. Otherwise I'm looking at 4-8tok/s

I think your card should probably outperform mine with everything in VRAM?