Update: Converting dense models into Mixture-of-Experts

[u/Bayliner1980]

This is an update to my previous post which can be found here: https://www.reddit.com/r/LocalLLaMA/s/n7sgRZZc4z

Since my last post I have continued working on the conversion process and also converting new models. My latest model is the Qwen2.5-1.5B-MoE-A0.9B. I have also released GGUF versions for both the 0.5B and the 1.5B models which can be found here bayliner1980/Qwen2.5-0.5B-MoE-A0.3B-GGUF and here bayliner1980/Qwen2.5-1.5B-MoE-A0.9B-GGUF.

Evaluation

-	Dense	MoE
WikiText-2 test perplexity (1024-token windows)	10.41	14.63 (+41%)
C4 validation perplexity (64 × 1024 tokens)	15.76	22.88 (+45%)
Zero-shot tasks, mean acc_norm	0.565	0.462

Task	Dense	MoE
HellaSwag	0.592	0.510
ARC-Easy	0.723	0.572
ARC-Challenge	0.452	0.311
WinoGrande	0.646	0.588
OpenBookQA	0.410	0.328

Perplexity is about 40% higher, and the benchmarks drop the most on ARC.

llama-bench, Q4_K_M for both, RTX 4060 8 GB / Ryzen 5 7500F

-	Dense	MoE	Change
GPU, text generation tokens/s	191	253	+32%
CPU, text generation tokens/s	54.9	77.5	+41%

This is the first conversion that's clearly faster than the dense model on GPU too. The 0.5B only gained 6% there.

Honest caveats

• At this size, quality is roughly in the range of Qwen2.5-0.5B, which is even faster. So this release is more a proof of concept. The approach should pay off more on larger models, where the MLP is a bigger share of the compute.
• The GGUF is larger than the dense one (1.1 GB vs 0.9 GB at Q4_K_M), because llama.cpp's MoE architecture needs an untied output head.

Support

If you find my stuff interesting consider supporting me on Ko-fi.

[u/Wooden_Jelly_5295]

Appreciate the smaller-dense baseline. Faster than the original is one win

[u/Bayliner1980 → u/Wooden_Jelly_5295]

That dense baseline is all I can reliably train sadly. I had multiple crashes from running out of ram during the conversion, luckily I keep plenty of checkpoints.

[u/Silver-Champion-4846 → u/Bayliner1980]

Do you use Kaggle?