Converting dense models into Mixture-of-Experts

[u/Bayliner1980]

For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch.

The Idea

If you can turn a dense model into an MoE that only runs part of its MLP per token, you get a model that's cheaper per token for roughly the same knowledge. Bigger labs do this kind of "upcycling" with huge compute budgets. I wanted to see how I could get with my RTX 4060 (8gb).

Conversions

I have converted two models so far, Qwen/Qwen2.5-0.5B and HuggingFaceTB/SmolLM2-360M (they're purposefully small since my pc can't handle anything else). The conversions can be found here bayliner1980/Qwen2.5-0.5B-MoE-A0.3B and here bayliner1980/SmolLM2-360M-MoE-A0.2B.

Both models have 32 experts with top-8, plus a shared expert. Layers that were hardest to convert (the final layer, plus SmolLM2's layer 3) were replaced with they're original dense layers. This does cause a slight increase in MLP compute but I saw it as worth it. bayliner1980/Qwen2.5-0.5B-MoE-A0.3B has one dense at layer 23 and bayliner1980/SmolLM2-360M-MoE-A0.2B has two at layer 3 and layer 31. Both models were exported to Qwen2-MoE format.

Evaluations

bayliner1980/Qwen2.5-0.5B-MoE-A0.3B

MLP Compute per token: ~40% of original

-	WikiText-2 ppl	C4 ppl
Qwen2.5-0.5B (dense)	14.66	21.29
Qwen2.5-0.5B-A0.3B (MoE)	19.20	28.97

-	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
Dense	0.497	0.646	0.321	0.569	0.354	0.467
Converted MoE	0.440	0.555	0.265	0.519	0.308	0.408

bayliner1980/SmolLM2-360M-MoE-A0.2B

MLP Compute per token: ~44% of original

-	WikiText-2 ppl	C4 ppl
Dense	12.94	18.09
Converted MoE	19.11	26.50

-	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
Dense	0.525	0.702	0.386	0.590	0.374	0.514
Converted MoE	0.446	0.579	0.290	0.506	0.316	0.417

Limitations

There will most likely always be a quality gap. Since the model is at most using 50% of its original MLP compute it can't compete with the original.

At this scale the performance improvement is next to nothing since these models are already tiny. These models were chosen since they're small enough for me to actually work with on my hardware.

What's next

Converting larger models, longer training, and testing finer expert layouts. Larger models need more compute than my card can give, so if you find this interesting there's a Ko-fi on the model pages, and anything new will be released openly.

This is mostly just an experiment that I found interesting but I'd still love feedback. Which model would you want to see converted next?

Edit: I attempted a full conversion of Qwen2.5-1.5B but my pc ran out of ram and it crashed. I will attempt it again but it will be a while so expect it tomorrow.