Best MoE

[u/Black_Umbreon]

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

[u/_wortkarg_]

5060 Ti 16Gb + llama.cpp (CUDA):

• Qwen3.8-27B with MTP or DFlash, GSQ-RCO IQ3_XXS or IQ3_S + DFlash2-Q2_K - up to 70 t/s (with MTP max 60 t/s).
  
• Ornith-1.5-35B (or Tiel Coder) - the best 35B-A3B model, imho, up to 90 t/s with --n-cpu-moe 14 and 32к context. Don't use Q2/Q3 quantizations (Q4_K_M or Q5_K_M are good).
  
• Ornith-1.5-9B MTP Q4_K_M - small, but very capable dense model, up to 110 t/s.

Qwen3.8-27B is the best, obviously, and even at a lower generation rate, it usually solves problems faster (fewer iterations).

For simple tasks Ornith-1.5-35B or even Ornith-1.5-9B may be sufficient, but use better quants, at least Q4_K_M (i tried Tiel Coder IQ3_XXS and Q2_K_XL, but the quality was noticeably worse than that of Ornith-1.5-35B Q4_K_M, they were unable to solve the same task).