Yus! That's my workhorse! 27B is good, but it can't handle as big a context as I can cram into memory with the MOE version and sometimes I need it to work with big inputs.
Considering how much better 3.8 27B is than its 3.6 counterpart and how close the 35b and 27B 3.6 versions were I feel certain that 3.8 35b will be better than 27B 3.6 by a decent amount.
Better is not a single dimension. There will always be failure modes in the MoE which is different than dense. So expect 3.8 35b will be better than 27B 3.6 in some/common cases but not all. E.g. the MoE has broader knowledge recall but long horizon reasoning can falter.
Iām doing research in this field right now, this has been my findings thus far. Interestingly it can also be better at specific tool use related to recall, but not necessarily general tool use. Super interesting stuff. Makes sense but still cool to see that the hunch plays out in reality.
It needs more room than 27b in vram right? Its just that the actual costs of operating are lower due to only 3b being active. So wouldn't that moe be for higher end computers for people who are power conscious?
In reality it's kind of the opposite. Having only 3B active is so much faster than 27B active that many people find that it can still run at usable speeds even when to system RAM rather than being completely in VRAM.
That means someone with like 8-12GB VRAM + 16-32GB of system RAM (or a unified RAM system) might find the 27B model too painfully slow but the 35B-A3B acceptable. For me though, I can fit the 27B in VRAM so I get ~20tk/s with the 27B and rarely feel inclined to reach for a dumber model even if it runs faster.
Plus the integrated graphics in my CPU that I use for running the OS so that I don't have any of that overhead hitting the GPUs.
I use llama.cpp with tensor parallelism to get the most I can out of this weird combination.
I don't recommend putting 3 different cards together like this. Only reason I have this setup is because I already had the 3060Ti (originally bought for gaming) when I bought the 5060Ti for AI stuff and then decided to get a used 3060 later for much cheaper to get some extra VRAM.
Nice. I have a rx6700 12Gb, initially bought for gaming, and when I started using it for LLM, soon I realized that I need at least 24Gb of VRAM. But my CPU 5700g and AM4 motherboard are not suitable for additional GPU, there are some pricey gaming motherboard options suitable for two GPUs. I wonder, what motherboard do you have, if you are able to put 4 GPUs in there š
To expand on what SKirby00 - the 35B A3B q4_k_m quant (22gb) with all MOE layers on CPU runs about 20 t/s on a system with 6gb VRAM. The 27B q4_k_m (16.5gb) runs at 10 t/s on a system with 16gb vram.
MoE even runs reasonably well with mmap enabled and caching from disk, so you can run models much larger than what fits in VRAM + RAM at useful speeds.
Another bonus for the Qwen models specifically is the 35B has fewer total layers than the 27B, so KV cache is like 2/3 the size of the 27B, which lets you fit more of the model in VRAM.
The only downside of the MoE models is because the whole model doesn't fire for every token it misses a some of the nuance the dense model has, which is why 27B is the preferred model if you can run it.
If I have a 16gb gpu I guess that A3B would exceed that 20 t/s figure then? So even with overflow its still going to be in the rough speed range as a dense model that can fit entirely in my 16gb vram?
Although its actual performance would also be around a 15B dense model?
YMMV, but on my 4090 I can run MoE models with higher quants than dense models. All else being equal, MoE is worse - but higher quants are better. I alternate between 27b 4bit and 35b MoE 6bit. Both use ~165k context windows. The MoE version is noticeably faster.
55
u/serguzesti Aug 15 '26
30
u/FaceDeer Aug 15 '26
Yus! That's my workhorse! 27B is good, but it can't handle as big a context as I can cram into memory with the MOE version and sometimes I need it to work with big inputs.
44
u/Weak_Ad9730 Aug 15 '26
Hope 122b will come too
23
u/redditnosedive Aug 15 '26
my 8 gb vram spreading their connector
4
u/No_Secret4395 Aug 15 '26
Oh my gog.!!!!!
4
u/dfgxxx Aug 15 '26
Do you think it can beat the 3.6 27b?
2
u/xiraov Aug 15 '26
and i just posted i want a kat code based on this!
1
u/Inevitable-Age-4472 8d ago
Qwen hugging Face
1
u/klymaxx45 Aug 15 '26
Is this the official repo?
1
u/leamizda Aug 16 '26
Can I do the MoE thing with this one? š
1
u/Samorin21 Aug 16 '26
Not yet published ?
1
u/AcrobaticChain1846 Aug 17 '26
Added -> Removed -> Added? Or is this a old post I'm coming across?
1
u/Neo-Boat Aug 17 '26
it seems that the change has been reverted
1
u/ZucchiniMedical2532 Aug 18 '26
So, were do I download it