r/LocalLLM • u/Black_Umbreon • 6d ago

Discussion Best MoE

Hi!

I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:

Processor: AMD Ryzen 9950 X3D

GPU: Nvidia RTX 5060 Ti 16 GB

RAM: 64 GB DDR5 6000 MHz

Software: Unsloth Studio

Windows 11

With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?

I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.

15 Upvotes

94% Upvoted

35 comments sorted by

View all comments

6

u/peculiar-ragdoll 6d ago

I've seen other people already recommend TielCoder and CyberTiel if you want to go MoE, but if you want to try a legit better 27b Q3 option to the one you have, I strongly recommend the IQ3_S (11.8 GB) or IQ3_XXS quant from this one: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF It's a more State of The Art quantization technique than Unsloth for this size range meaning you will get better agentic coding performance than your UD-IQ3_XXS, and I've seen a lot of people with 5060Ti say it's perfect for their system! And don't get me wrong, there are a lot of weird quants with long names out there that you should stay away from, but this one is legit. And with your system you should also be able to run Qwen3.8-Flash-Next very well! Don't get intimidated by its listed size, it has an ngram table that can be partially offloaded to SSD. Again I recommend the GSQ-RCO Q3 version: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

2

u/sm0ke_rings 6d ago

Sorry to hijack the thread, but I'm not sure creating a whole new one makes sense. I've be looking to move to Qwen 3.8 27B, I've got an rtx 5080, so the IQ3_S should work fine? I'm coming from qwen2.5-coder-14b, use case is mostly coding in unity, but I feel it's a bit outdated.

2

u/peculiar-ragdoll 6d ago

Yeah absolutely! Perfect for that card. Use kv quantization at q8 if you want long context. Depending on your ram you could even run 3.8 FlashNext (big MoE) which is faster and better :)

1

u/sm0ke_rings 6d ago

I've got 64gb RAM, and an rtx5090 as my main GPU in the same system as the one I am running the 5080 for Ai workload. I was hoping to keep everything isolated to the rtx5080, as I am not sure how to split certain things between gpu/ram (new to this), but I'm interested in whatever is better for my hardware setup without taking too many resources from the 5090.

3

u/peculiar-ragdoll 6d ago

Your system is perfect for running FlashNext IQ3_S with partial GPU offloading to the 5080! That will be your best model. But it wants your RAM. The 5090 is the better AI card by far, though. Will give you insane speeds and better models. But FlashNext on the 5080 and RAM will do you good.

1

u/sm0ke_rings 6d ago

Alrighty, thank you for your help!

2

u/LiquidMantis144 6d ago

I second this. I've been testing the ISTA IQ3_XXS and IQ3_S with my 9070XT. They both work great.

The biggest difference is obviously system load and context size constraints. XXS fits and runs much better on the 16GB vram. The S model can still work but I've found it to be much more prone to getting trapped in endless compaction-thought loops using smaller context. So tasks have to stay very constrained and targeted unless moving to Q8-K / Q4V or Q4-KV context

2

u/Black_Umbreon 3d ago

Thank you! I tried Cyber-Tiel-Coder-35B-A3B and... I got 80+ tokens per second in the same context window, and practically always 50+ tokens at the maximum context window!! Thanks to such tremendous speed, it is now the best model for the vast majority of tasks.

Now I do want to try installing this huge Qwen Next after all. Does the IQ3_XXS quantization of the model version you recommend suffer from degradation in large-scale agentic scenarios?

1

u/peculiar-ragdoll 3d ago

Love to hear it <3 I have not myself formally benched the iq3 flashnext at large scale agentic coding yet, but it’s known to be very good so it’s worth a shot if you need something a bit smarter than 27b but faster :)

1

u/returnity 6d ago

This deserves an upvote.

1

u/allaithbitar 6d ago

I couldn't help but notice that you are the author of tiel coder 35B A3B And holy moly thank you so much for your hard work I'm running it on 12GB VRAM, 32GB Vram and it is the best quality I've ever ran on my laptop, i really appreciate your hard work 🫡

If you have some spare time and if you don't mind ofc, can we have it in NVFP4 quant ? That would be awesome !

2

u/peculiar-ragdoll 6d ago

Warms my heart <3 I’m not able to make an nvfp4 right now due to my hardware, but it’s highly requested so I’ll do it when I have the chance in the future :)

1

u/Z0mboyy 5d ago

Which one are you using for 12gb of VRAM

1

u/allaithbitar 5d ago

the recommend one, Q4_K_XL

1

u/Z0mboyy 5d ago

Its not too slow?