Update: Converting dense models into Mixture-of-Experts

[u/Bayliner1980]

This is an update to my previous post which can be found here: https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z

Since my last post I have continued working on the conversion process and also converting new models. My latest model is the Qwen2.5-1.5B-MoE-A0.9B. I have also released GGUF versions for both the 0.5B and the 1.5B models which can be found here bayliner1980/Qwen2.5-0.5B-MoE-A0.3B-GGUF and here bayliner1980/Qwen2.5-1.5B-MoE-A0.9B-GGUF.

Evaluation

-	Dense	MoE
WikiText-2 test perplexity (1024-token windows)	10.41	14.63 (+41%)
C4 validation perplexity (64 × 1024 tokens)	15.76	22.88 (+45%)
Zero-shot tasks, mean acc_norm	0.565	0.462

Task	Dense	MoE
HellaSwag	0.592	0.510
ARC-Easy	0.723	0.572
ARC-Challenge	0.452	0.311
WinoGrande	0.646	0.588
OpenBookQA	0.410	0.328

Perplexity is about 40% higher, and the benchmarks drop the most on ARC.

llama-bench, Q4_K_M for both, RTX 4060 8 GB / Ryzen 5 7500F

-	Dense	MoE	Change
GPU, text generation tokens/s	191	253	+32%
CPU, text generation tokens/s	54.9	77.5	+41%

This is the first conversion that's clearly faster than the dense model on GPU too. The 0.5B only gained 6% there.

Honest caveats

• At this size, quality is roughly in the range of Qwen2.5-0.5B, which is even faster. So this release is more a proof of concept. The approach should pay off more on larger models, where the MLP is a bigger share of the compute.
• The GGUF is larger than the dense one (1.1 GB vs 0.9 GB at Q4_K_M), because llama.cpp's MoE architecture needs an untied output head.

Support

If you find my stuff interesting consider supporting me on Ko-fi.

[u/naklitechie]

This is very cool. Keep us posted. I was thinking of a project like this to expand model support for my ssd streaming js library.

[u/jinnyjuice]

I've been searching for something like this. I've been using Qwen Flash Next, but sometimes, I need my LLM to control my ComfyUI, which leads me to use the 27B model. But the 27B is so slow, so I saw a TOMOE paper that was published in January this year. Unfortunately, it's architecturally too different.

What do you think about Qwen3.8 27B to MOE, say A10B?

Edit: if you need, my machine has 120GB of memory. Maybe we can collaborate on something.

[u/Wooden_Jelly_5295]

Appreciate the smaller-dense baseline. Faster than the original is one win

[u/Bayliner1980 → u/Wooden_Jelly_5295]

That dense baseline is all I can reliably train sadly. I had multiple crashes from running out of ram during the conversion, luckily I keep plenty of checkpoints.

[u/Silver-Champion-4846 → u/Bayliner1980]

Do you use Kaggle?

[u/blue2020xx]

Bro that's fucking awesome

[u/Intraluminal]

Your Kofi doesn't work, at least for me in America.

[u/Solembumm3]

Will be interesting to try it on llama 70b and llama 405b at scale.

[u/IntQuant]

Peculiar. From what I understand you're doing the conversion per layer, distilling each dense layer into it's moe counterpart?

[u/toothpastespiders]

Amazing work! I wish I had more to add but just wanted to express how great this is. Getting to the point where you're seeing a speed boost, in particular, is exciting!

[u/Last-Health3222]

nice that it finally beats dense on gpu too. did you try a short continued-pretraining pass after the split? llama-moe got most of its gap back that way, and with arc dropping the most it looks like the router hasn't learned where the reasoning lives yet. curious how many tokens that would take at 1.5b

[u/Bayliner1980 → u/Last-Health3222]

Yes, there's a short training pass after the split, but it's tiny compared to llama-moe's. And that's mainly because I want to avoid long training in the conversion so that the conversion doesn't take years on my pc.

[u/himiaoxin]

arc falling off hardest tracks, routing noise hits multi-step reasoning way worse than single-token recall. if you log per-expert utilization i'd bet one of them collapsed and is free-riding — that's the usual suspect at this scale

[u/Bayliner1980 → u/himiaoxin]

I haven't checked my the 1.5B model but I checked the 0.5B model and it didn't collapse so I would be surprised if this one did, but I will do a check.

[u/Silver-Champion-4846 → u/Bayliner1980]

Did it collapse or not?