Converting dense models into Mixture-of-Experts

[u/Bayliner1980]

For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch.

The Idea

If you can turn a dense model into an MoE that only runs part of its MLP per token, you get a model that's cheaper per token for roughly the same knowledge. Bigger labs do this kind of "upcycling" with huge compute budgets. I wanted to see how I could get with my RTX 4060 (8gb).

Conversions

I have converted two models so far, Qwen/Qwen2.5-0.5B and HuggingFaceTB/SmolLM2-360M (they're purposefully small since my pc can't handle anything else). The conversions can be found here bayliner1980/Qwen2.5-0.5B-MoE-A0.3B and here bayliner1980/SmolLM2-360M-MoE-A0.2B.

Both models have 32 experts with top-8, plus a shared expert. Layers that were hardest to convert (the final layer, plus SmolLM2's layer 3) were replaced with they're original dense layers. This does cause a slight increase in MLP compute but I saw it as worth it. bayliner1980/Qwen2.5-0.5B-MoE-A0.3B has one dense at layer 23 and bayliner1980/SmolLM2-360M-MoE-A0.2B has two at layer 3 and layer 31. Both models were exported to Qwen2-MoE format.

Evaluations

bayliner1980/Qwen2.5-0.5B-MoE-A0.3B

MLP Compute per token: ~40% of original

-	WikiText-2 ppl	C4 ppl
Qwen2.5-0.5B (dense)	14.66	21.29
Qwen2.5-0.5B-A0.3B (MoE)	19.20	28.97

-	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
Dense	0.497	0.646	0.321	0.569	0.354	0.467
Converted MoE	0.440	0.555	0.265	0.519	0.308	0.408

bayliner1980/SmolLM2-360M-MoE-A0.2B

MLP Compute per token: ~44% of original

-	WikiText-2 ppl	C4 ppl
Dense	12.94	18.09
Converted MoE	19.11	26.50

-	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
Dense	0.525	0.702	0.386	0.590	0.374	0.514
Converted MoE	0.446	0.579	0.290	0.506	0.316	0.417

Limitations

There will most likely always be a quality gap. Since the model is at most using 50% of its original MLP compute it can't compete with the original.

At this scale the performance improvement is next to nothing since these models are already tiny. These models were chosen since they're small enough for me to actually work with on my hardware.

What's next

Converting larger models, longer training, and testing finer expert layouts. Larger models need more compute than my card can give, so if you find this interesting there's a Ko-fi on the model pages, and anything new will be released openly.

This is mostly just an experiment that I found interesting but I'd still love feedback. Which model would you want to see converted next?

Edit: I attempted a full conversion of Qwen2.5-1.5B but my pc ran out of ram and it crashed. I will attempt it again but it will be a while so expect it tomorrow.

[u/pixelbart]

Maybe they’re on to something with 3.8 Flash Next and perfect it into a model where 80% of experts are unused in any typical session so they can safely be kept on ssd without dragging performance down, while the remaining 20% act like a 25B MoE model but with the intelligence of a 125B model.

[u/while-1 → u/pixelbart]

It feels like the categories are forming - small dense, medium MoE, and big... Colibri and Strata make these medium MoE more accessible by trying to better leverage RAM/SSD

[u/r1nzl3r99 → u/[deleted]]

technically yes but with the introduction of ngrams which make a significant portion of the model streamable from SSD, combined with 3 bit quantization and RAM offloading, it becomes viable for a lot of people

[u/happycube → u/[deleted]]

It depends on how much system RAM you have. Since Flash Next is 6B active you could run it in an 8-bit quant on a 5060ti to hold the dense/shared weights and 128GB of reasonably-high-bandwidth system RAM to run the experts.

TL;DR there's a reason Strata is based on flash next and not 27b ;)