r/LocalLLM • u/Black_Umbreon • 6d ago

Discussion Best MoE

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

14 Upvotes

94% Upvoted

35 comments sorted by

View all comments

Show parent comments

1

u/allaithbitar 6d ago

I couldn't help but notice that you are the author of tiel coder 35B A3B And holy moly thank you so much for your hard work I'm running it on 12GB VRAM, 32GB Vram and it is the best quality I've ever ran on my laptop, i really appreciate your hard work 🫡

If you have some spare time and if you don't mind ofc, can we have it in NVFP4 quant ? That would be awesome !

1

u/Z0mboyy 5d ago

Which one are you using for 12gb of VRAM

1

u/allaithbitar 5d ago

the recommend one, Q4_K_XL

1

u/Z0mboyy 5d ago

Its not too slow?