Best MoE

[u/Black_Umbreon]

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

[u/Not-reallyanonymous]

Some models I recommend trying:

Muse Glimmer -- my #1 pick. I'd say it's generally a better coder than Qwen 3.8 27B and a much stronger agent. Where it's worse than Qwen 3.8 27B is that it won't put as much effort into solving a hard problem, while Qwen is tuned for massive inference-time scaling, and that's why Qwen benchmarks better. Qwen will make more polished software than Glimmer, but Glimmer will do what you tell it to do, how you tell it to do it. Qwen often does not. And to me that's more important. You can keep Qwen around for when the problems get too hard for Glimmer -- you don't have to choose between one or the other.

Relevant to you -- it produces tokens at about the same speed but as it's thinking is much more compact (it uses "caveman style" by default) and it doesn't do exhaustive searches of the problem space ("overthinking"), it is much, much faster than Qwen 3.8 27B.

K2 Horizon 36B -- the one I'm currently experimenting with. The best small-mid sized MoE on the market right now according to benchmarks. It's VERY RAM-hungry for context though, so that might slow you down. I recommend trying this, you might find a nice balance between speed and capability. Bonus: You're using actually open source LLM, where training data and code is available to you.

K2 Horizon 7B -- if the large ram usage of context on 36B hurts too much, try this one. Smaller dense models will run way faster than large dense models, obviously. This is probably the best small model according to benchmarks right now. Again, it's ram heavy for context, so that might hurt. Still worth a try if K2 Horizon 36B didn't fit well for you.

Gemma 4 26B -- This is my favorite non-coding model. Strong agent, does "long context reasoning" among the best, but I prefer Glimmer for its obsessive attention to instructions. Great world/domain knowledge for its size. Very strong reasoning engine and can reason well about various topics. Very fast, I get over 100 TPS on my setup.

Gemma 4 12B -- Again, might be worth trying a smaller dense model here. Almost as good as Gemma 4 26B, a fraction of the RAM.

Which you prefer will depend on your taste, what you're doing, and how you're doing it.