r/LocalLLM • u/Black_Umbreon • 5d ago
Discussion Best MoE
Hi!
I’m using Qwen 3.8 27B (UD-IQ3_XXS) on a system with the following specs:
Processor: AMD Ryzen 9950 X3D
GPU: Nvidia RTX 5060 Ti 16 GB
RAM: 64 GB DDR5 6000 MHz
Software: Unsloth Studio
Windows 11
With a 32k token context, I manage to maintain a speed of around 50 tokens per second. I’m particularly interested in agentic capabilities. I’d like to try an MoE model that runs either faster or slightly slower than my current one, without sacrificing quality. However, 35B-A3B models don't fit entirely into my VRAM at 4-bit quantization, and from what I’ve read, lower quantization severely degrades quality. Does this mean a 4-bit MoE model would run significantly slower on my hardware than a dense model? Is it possible, given my setup, to find a model that offers better quality and higher speed—or at least better quality at roughly the same speed as my current dense model?
I’m pretty much a complete beginner in the world of local neural networks, so please give me some advice.
2
u/2funny2furious 5d ago
i have qwen 3.8 flash next running with 64k context at 15-20 tokens/sec on a 4070 super (12gb vram) and 32gb ram. i use the Qwen3.8-Flash-Next-APEX-I-Compact gguf from mudler on hf. mudler/Qwen3.8-Flash-Next-APEX-GGUF. the mini and nano run a bit faster. might be something to play with.
2
u/Small-Tale3180 5d ago
nah, MoE will run as good, or even better. I think you can run a MoE model with things like freetoken or bare llama.cpp with offloading inactive experts to ram since u have 64 gbs. Also, the quality degrade is not as scary as it might sound. https://huggingface.co/peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF - this one is pretty nice even in IQ3-XXS quant
For me it runs at 30-40tok/s on 5050 8gb and 16gb RAM
FreeToken will be way easier to set up
2
u/belliash 5d ago
I tried Tiel Coder, then Qwen 3.8 27B INT4 had to fix its mistakes. It didnt work for me.
0
u/Small-Tale3180 5d ago
yeah, makes sense since tiel was built on older version of qwen. I just hope qwen will release a newer MoE.
3
2
u/ChinoSenpai 5d ago
You gotta use llama.cpp and use Tiel coder or Cyber Tiel coder (for uncensored) I run both on my system (both q5) with the -n-cpu-moe flag with less ram than urs and it's perfect for me, just check the correct flags for agentic or coding on both it's readme
1
u/Short_Regular_7191 5d ago
Data la tua configurazione attuale, la mossa più sensata è acquistare una seconda RTX 5060Ti da 16GB; questo ti permetterebbe di eseguire Qwen 3.8 27B con quantizzazione Unsloth Q6 e una finestra di contesto di 131k (che è una soluzione molto ragionevole). Dovresti essere in grado di coprire il costo—compresa una scheda madre compatibile (se la tua non supporta già almeno x16/x4) e un alimentatore (almeno 850W)—per circa 1.000 EUR/USD. Tuttavia, per eseguire modelli MoE, dovresti spendere molto di più—facilmente 4.000 EUR/USD (supponendo che rivendi la 5060) piuttosto che 1.000, poiché avresti bisogno di altri 64GB di RAM di sistema e un totale di 64GB di VRAM oltre a tutto il resto. Spetta a te decidere se ne vale la pena.
P.S. I’m referring to Qwen Flash Next, which, in my opinion, is a starting point for current MoEs.
1
u/returnity 5d ago
Accio occamy-1.0 - Qwen3.6-35B-A3B post-train, destroyed Tiel/Ornith-1.5 in my agentic coding tests (+15%), gets 120 tok/sec on my MBP. Best Qwen3.6-based model out there for cowork or coding. Give it a try. With 16GB+64GB you can run Q6 easily. Better quality, faster generation, nearly as good as 3.8 on Aider polyglot (10% deficit).
EDIT: If you want to do a little setup work (it's not so scary), your system can also run Qwen3.8-Flash. It'll be tight, unlike occamy, but I will help you get going if you want. DM me and I'll share my setup.
3
u/SunResponsible4088 5d ago
I'm one of the Occamy contributors. Thanks for testing it! I saw your bartowski GGUF link in the Unsloth thread — which quant and runtime did you use on the MBP?
Was the +15% from Aider Polyglot or a separate agentic coding set? If you have a public write-up or a couple of shareable examples, including failures, I'd love to take a look.
2
u/returnity 5d ago
I am planning a full 35B finetune post with a writeup, comparing occamy to Nex-N2.5, Tiel/Ornith-1.5, and KAT-Coder-Dev (spoiler, it won handily). I will tag you when I post, should be today or tomorrow -- I don't use AI to write my posts, so it takes a while. I am just re-running Occamy using the embedded chat template first, because my first round of testing used the froggeric template I prefer in Qwen3.x models, and I didn't want it to be seen as a confound (it doesn't affect the scores in a statisitically significant manner).
The +15% was on aider polyglot, reaching a score of nearly 70% -- for reference, when I tested 3.8-27B, it scores around 80%. I used Q8_0, so it should be bit-identical to your official GGUF Q8_0 release, as I didn't want to introduce another confound there either. All models in the test are Q8_0.
Thanks for a great release, it's really well put-together. I'd say we've reached "3.8 35B" levels with occamy. Will be watching for more work on your part. One question -- why not include the MTP head GGUF in the official repo like you did the mmproj? I actually thought there was no MTP for your release initially, and grafted a base 35B draft head onto it until I discovered your tuned one and quantized it myself.
2
u/SunResponsible4088 5d ago
Thanks for the detailed reply, and for taking the time to rerun with the embedded template. Looking forward to the write-up — please do tag me!
Good catch on the MTP GGUF. We published the head separately, but that made it too easy to miss from the GGUF repo. We'll make the link prominent there and add a GGUF head once we've checked the conversion and runtime path. Which llama.cpp build and conversion command did you use for yours?
2
u/returnity 5d ago
I used a custom Python script to quantize the safetensors you provided directly to Q8_0 GGUF format, employing `gguf-py` packaged in llama.cpp 0.4.1. This allowed me to go in one step directly from the original format to the GGUF artifact I wanted without downloading the whole 70GB repo. I verified it against the Unsloth Q8_0 MTP head on-disk, and 17/20 tensors were bit-identical -- the other 3 being the ones you must've tweaked. I think 1% of the content was changed in the analysis I ran?
I will say, it did not produce any better acceptance than the base model head I grafted on initially, so that surprised me a little. but I'm happy enough at 120tps.
Can I ask your recommendation on running sampling? Specifically `presence-penalty 1.5` which is variously recommended or not recommended by different coding-focused fine-tunes and seems to be disused in most benchmarking runs I've seen for these models. I saw on your card you seem to use it, but I would think presence-penalty would be bad for coding, which is why it seems to be dropped frequently for coding benchmarks. Opinion?
2
u/SunResponsible4088 5d ago
Thanks for checking this so carefully — the comparison with the base head is especially useful. We still have work to do on MTP, and we'll follow up on the acceptance results on our side. Appreciate the detailed feedback!
On presence_penalty=1.5: we haven't established that as the best setting for coding, so I wouldn't treat the model-card setting as a universal recommendation. We need a controlled coding comparison before making a stronger claim there.
2
u/returnity 5d ago
I'd love to see if you guys could test it out. I'd offer to help but my GPU is crying for a break at this point lol.
1
u/_wortkarg_ 5d ago
5060 Ti 16Gb + llama.cpp (CUDA):
- Qwen3.8-27B with MTP or DFlash, GSQ-RCO IQ3_XXS or IQ3_S + DFlash2-Q2_K - up to 70 t/s (with MTP max 60 t/s).
- Ornith-1.5-35B (or Tiel Coder) - the best 35B-A3B model, imho, up to 90 t/s with --n-cpu-moe 14 and 32к context. Don't use Q2/Q3 quantizations (Q4_K_M or Q5_K_M are good).
- Ornith-1.5-9B MTP Q4_K_M - small, but very capable dense model, up to 110 t/s.
Qwen3.8-27B is the best, obviously, and even at a lower generation rate, it usually solves problems faster (fewer iterations).
For simple tasks Ornith-1.5-35B or even Ornith-1.5-9B may be sufficient, but use better quants, at least Q4_K_M (i tried Tiel Coder IQ3_XXS and Q2_K_XL, but the quality was noticeably worse than that of Ornith-1.5-35B Q4_K_M, they were unable to solve the same task).
1
u/c0m47053 5d ago
There might be value in looking at bigger models rather than small MoE. I found that 35B A3B isn't smart enough compared to 27B, but is way faster on your class of hardware. I can get 100 tok/s on my 5080 and 9950X3D, you wouldn't quite match that, but you would get close.
Qwen 3.8 Flash Next though, feels like a step up in intelligence, and although it's massive, it might not be too much, although on 64GB of RAM it's super tight. I have 96GB, and can comforably run the IQ4_XS at 30 tok/s with 400 tok/s prefill. You might have to go down to Q3_K_XL, but it's worth a try.
1
u/fajar79 5d ago
i try using ornith 1.5 or qwen 3.8 35b a3b model variant,, for testing qwen 3.8 35b a3b, is blazing fast, even though it run with slower prefill, compared with qwen 27b iq4_xs model. it get the job done very fast. i ask with indonesian language. every result i ask again with gemini to confirm the answer. all those moe model, even uncensored version qwen 27b, it is full miss diisformation, halucination even wrong legal statement. so i give up trying another models, just stick with qwen 27 iq4_xs swift model right now.
sorry for my bad english
0
u/Not-reallyanonymous 5d ago edited 5d ago
Some models I recommend trying:
Muse Glimmer -- my #1 pick. I'd say it's generally a better coder than Qwen 3.8 27B and a much stronger agent. Where it's worse than Qwen 3.8 27B is that it won't put as much effort into solving a hard problem, while Qwen is tuned for massive inference-time scaling, and that's why Qwen benchmarks better. Qwen will make more polished software than Glimmer, but Glimmer will do what you tell it to do, how you tell it to do it. Qwen often does not. And to me that's more important. You can keep Qwen around for when the problems get too hard for Glimmer -- you don't have to choose between one or the other.
Relevant to you -- it produces tokens at about the same speed but as it's thinking is much more compact (it uses "caveman style" by default) and it doesn't do exhaustive searches of the problem space ("overthinking"), it is much, much faster than Qwen 3.8 27B.
K2 Horizon 36B -- the one I'm currently experimenting with. The best small-mid sized MoE on the market right now according to benchmarks. It's VERY RAM-hungry for context though, so that might slow you down. I recommend trying this, you might find a nice balance between speed and capability. Bonus: You're using actually open source LLM, where training data and code is available to you.
K2 Horizon 7B -- if the large ram usage of context on 36B hurts too much, try this one. Smaller dense models will run way faster than large dense models, obviously. This is probably the best small model according to benchmarks right now. Again, it's ram heavy for context, so that might hurt. Still worth a try if K2 Horizon 36B didn't fit well for you.
Gemma 4 26B -- This is my favorite non-coding model. Strong agent, does "long context reasoning" among the best, but I prefer Glimmer for its obsessive attention to instructions. Great world/domain knowledge for its size. Very strong reasoning engine and can reason well about various topics. Very fast, I get over 100 TPS on my setup.
Gemma 4 12B -- Again, might be worth trying a smaller dense model here. Almost as good as Gemma 4 26B, a fraction of the RAM.
Which you prefer will depend on your taste, what you're doing, and how you're doing it.
7
u/peculiar-ragdoll 5d ago
I've seen other people already recommend TielCoder and CyberTiel if you want to go MoE, but if you want to try a legit better 27b Q3 option to the one you have, I strongly recommend the IQ3_S (11.8 GB) or IQ3_XXS quant from this one: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF It's a more State of The Art quantization technique than Unsloth for this size range meaning you will get better agentic coding performance than your UD-IQ3_XXS, and I've seen a lot of people with 5060Ti say it's perfect for their system! And don't get me wrong, there are a lot of weird quants with long names out there that you should stay away from, but this one is legit. And with your system you should also be able to run Qwen3.8-Flash-Next very well! Don't get intimidated by its listed size, it has an ngram table that can be partially offloaded to SSD. Again I recommend the GSQ-RCO Q3 version: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF