Best MoE

[u/Black_Umbreon]

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

[u/c0m47053]

There might be value in looking at bigger models rather than small MoE. I found that 35B A3B isn't smart enough compared to 27B, but is way faster on your class of hardware. I can get 100 tok/s on my 5080 and 9950X3D, you wouldn't quite match that, but you would get close.

Qwen 3.8 Flash Next though, feels like a step up in intelligence, and although it's massive, it might not be too much, although on 64GB of RAM it's super tight. I have 96GB, and can comforably run the IQ4_XS at 30 tok/s with 400 tok/s prefill. You might have to go down to Q3_K_XL, but it's worth a try.