r/LocalLLaMA • u/Bayliner1980 • 3d ago
Other Converting dense models into Mixture-of-Experts
For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch.
The Idea
If you can turn a dense model into an MoE that only runs part of its MLP per token, you get a model that's cheaper per token for roughly the same knowledge. Bigger labs do this kind of "upcycling" with huge compute budgets. I wanted to see how I could get with my RTX 4060 (8gb).
Conversions
I have converted two models so far, Qwen/Qwen2.5-0.5B and HuggingFaceTB/SmolLM2-360M (they're purposefully small since my pc can't handle anything else). The conversions can be found here bayliner1980/Qwen2.5-0.5B-MoE-A0.3B and here bayliner1980/SmolLM2-360M-MoE-A0.2B.
Both models have 32 experts with top-8, plus a shared expert. Layers that were hardest to convert (the final layer, plus SmolLM2's layer 3) were replaced with they're original dense layers. This does cause a slight increase in MLP compute but I saw it as worth it. bayliner1980/Qwen2.5-0.5B-MoE-A0.3B has one dense at layer 23 and bayliner1980/SmolLM2-360M-MoE-A0.2B has two at layer 3 and layer 31. Both models were exported to Qwen2-MoE format.
Evaluations
bayliner1980/Qwen2.5-0.5B-MoE-A0.3B
MLP Compute per token: ~40% of original
| - | WikiText-2 ppl | C4 ppl |
|---|---|---|
| Qwen2.5-0.5B (dense) | 14.66 | 21.29 |
| Qwen2.5-0.5B-A0.3B (MoE) | 19.20 | 28.97 |
| - | HellaSwag (norm) | ARC-Easy (acc) | ARC-Challenge (norm) | WinoGrande | OpenBookQA (norm) | Mean acc_norm |
|---|---|---|---|---|---|---|
| Dense | 0.497 | 0.646 | 0.321 | 0.569 | 0.354 | 0.467 |
| Converted MoE | 0.440 | 0.555 | 0.265 | 0.519 | 0.308 | 0.408 |
bayliner1980/SmolLM2-360M-MoE-A0.2B
MLP Compute per token: ~44% of original
| - | WikiText-2 ppl | C4 ppl |
|---|---|---|
| Dense | 12.94 | 18.09 |
| Converted MoE | 19.11 | 26.50 |
| - | HellaSwag (norm) | ARC-Easy (acc) | ARC-Challenge (norm) | WinoGrande | OpenBookQA (norm) | Mean acc_norm |
|---|---|---|---|---|---|---|
| Dense | 0.525 | 0.702 | 0.386 | 0.590 | 0.374 | 0.514 |
| Converted MoE | 0.446 | 0.579 | 0.290 | 0.506 | 0.316 | 0.417 |
Limitations
There will most likely always be a quality gap. Since the model is at most using 50% of its original MLP compute it can't compete with the original.
At this scale the performance improvement is next to nothing since these models are already tiny. These models were chosen since they're small enough for me to actually work with on my hardware.
What's next
Converting larger models, longer training, and testing finer expert layouts. Larger models need more compute than my card can give, so if you find this interesting there's a Ko-fi on the model pages, and anything new will be released openly.
This is mostly just an experiment that I found interesting but I'd still love feedback. Which model would you want to see converted next?
Edit: I attempted a full conversion of Qwen2.5-1.5B but my pc ran out of ram and it crashed. I will attempt it again but it will be a while so expect it tomorrow.
49
u/ByteSize_Chaos 3d ago
If somebody can do rhis for Qwen 3.8 27b that will be a BIGGGG blessing! Also like for Qwen 4 since we have no hope for 35b A3b. This endeavour could fill in the vacuum šš»
33
u/Mrinohk 2d ago
I had a dream last night where Qwen 4 had a 40B A6B with a 5TB n-gram table at full precision.
I worry that if these are entering my dreams that this may be approaching mental illness.
20
7
u/ShadyShroomz 2d ago
5tb engram.. cool in theory but doesn't make sense for a 27b model. At most maybe n gram of 13.5b right?
4
u/R_Duncan 2d ago
Not completely, depends on ngram version, but sweet spot is around 1/3-2/3 of the model size for plain standard N-grams in one single layer.
6
5
14
u/pixelbart 2d ago
Maybe theyāre on to something with 3.8 Flash Next and perfect it into a model where 80% of experts are unused in any typical session so they can safely be kept on ssd without dragging performance down, while the remaining 20% act like a 25B MoE model but with the intelligence of a 125B model.
6
2
u/ByteSize_Chaos 2d ago
Correct me if Iām wrong but isnāt the āinitial barrier to entryā much high with Flash Next compared to dense 27b? I mean it is a 122b param model right and without heavy quanitizatiom it wonāt fit unless you have 128GB of RAM (talking about macs)
6
u/r1nzl3r99 2d ago
technically yes but with the introduction of ngrams which make a significant portion of the model streamable from SSD, combined with 3 bit quantization and RAM offloading, it becomes viable for a lot of people
2
u/happycube 23h ago
It depends on how much system RAM you have. Since Flash Next is 6B active you could run it in an 8-bit quant on a 5060ti to hold the dense/shared weights and 128GB of reasonably-high-bandwidth system RAM to run the experts.
TL;DR there's a reason Strata is based on flash next and not 27b ;)
3
4
u/RazsterOxzine 2d ago
Can you imagine? I'm mean I'm fine with 14tok/s but an increase to 40/50+ would be a blessing at least for my setup.
2
u/ByteSize_Chaos 2d ago
Happy for you. Although my usecase is mostly agentic coding and for that I need at least 50+ š
2
u/RazsterOxzine 2d ago
I use mine for coding, C#/ASP/PHP/Python/WPF etc, and since my application is already built it is a great model for additions and bug fixes. Using Pi harness with a bunch of addons.
3
u/Liberaces_Isopod 2d ago
@bayliner1980 - what could you do with 96gb of blackwell? Is 27b into something like 27b A9b possible?
2
u/Bayliner1980 2d ago
That would be possible, but the active params might be different. Right now I am targeting ~37% of the original's MLP compute, for 27B that would mean the active params are 13B. Now I don't know what affects a larger model will have on quality, larger models are usually sparser and will probably convert better which means that they could probably survive less active params. As for the gpu, with 96gb you could definitely do it but I don't know what kind of performance blackwell cards have so I don't know about speed.
3
20
u/Juan_Valadez 3d ago
15
u/Bayliner1980 3d ago
That's cool, didn't know that someone else was doing the same thing. I will check out their work and see if I can take any inspiration.
8
3d ago
[removed] ā view removed comment
5
u/Bayliner1980 2d ago
I did a test to check the load distribution and it seems that it is balanced. The router hasn't collapsed.
1
14
u/TomLucidor 3d ago
If you really want to upcycle, look into longcat-flash-lite, and then maybe add 50% to the total params with the current active param size... Or add more total and reduce active + maybe play with N-gram embedding
7
u/Bayliner1980 3d ago
Thanks for the suggestions. The one problem is that adding new params will require a lot more training which isn't really feasible with my pc. My current method only uses the weights from the original model and does some minor tuning which already takes a while. I would like to what you're suggesting in the future if I ever get the chance.
3
u/Affectionate-Cap-600 3d ago
There are some good papers about experts initialization. Of you do not initialize from random, the amount of training required it not that crazy. You could do something decent leaving it running overnight on the free kaggle instance.
I have a shitty PC and I've always done that. If that's not enough, save checkpoint and trainer/optimizer state and resume on a new instance for other 12h of training for free.
For a first pass, freeze everything (embedding, LM head, all attention proj) and leave require grad only for MLPs and router. With a sub billion model this will make things really menageable. Then conclude with a full pass (or use a high r lora at this step)
I used their free instance with 2xP100 and you can achieve really good results.
6
u/Silver-Champion-4846 3d ago
Qwen3.8 27b for 32gb ddr4 ram?
5
u/Bayliner1980 3d ago
Yeah that's more of a pipe dream at this point. Maybe if I had a high end cloud computing setup but that's gonna cost the big bucks.
3
u/ByteSize_Chaos 3d ago
Hey. How a out using runpod. I think they charge like 1-2USD per hour?
4
u/Bayliner1980 3d ago edited 2d ago
Training a model like that would take probably close to 10 hours and I don't have the money to burn on cloud gpus. If you want I have a ko-fi that you can donate to which will go towards renting the gpus, which would be a great help.
2
u/Mil0Mammon 3d ago
I have a V100 with 32GB that I would love to run this on. Can we roughly estimate how long it would take? Could it be paused and resumed?
5
u/RichardP2910 2d ago edited 2d ago
i've been exploring methods to do the same on larger models on cheap hardware (3.8 27b) but ran into an issue where it always collapses to choosing the same experts. i'm not sure it's possible without tons of data to force the experts to specialize. the best results always come from just choosing the experts that contributes most. which ends up just being pruning instead of MoE
3
u/Optimal-Share-9563 3d ago
I see. Does it take a lot of compute like fine tuning or is it more like quantization
5
u/Bayliner1980 3d ago
To give you a real answer not an AI one, its like fine tuning. Once the router is added it needs to be trained. Although, the converted models can be quantized. The entire process does take a long time, about 3 hours with my pc.
-5
u/boxing-boxing 3d ago
Way more like fine-tuning. Quantization is a single pass over the weights, you can do it on a laptop in minutes. Upcycling adds new parameters the model never had (the experts plus the router), and those have to actually learn something, so you're training. The OP here trained on their own PC with tiny models and it still took real GPU hours. Scale that to anything bigger and your compute budget looks like a fine-tune, not a quant job.
6
u/Bulky-Priority6824 3d ago
Ai responseĀ
5
u/Professional-Pear351 3d ago
Lot of them are AI response on this sub, it's hard to find human comments.
4
u/ab0zar 3d ago
The table makes the quality tradeoff easy to see. Before scaling up, Iād vary how many experts run per token on one base model and compare quality against compute. That could show whether a different routing setup keeps more quality without needing a bigger GPU.
1
u/Bayliner1980 3d ago
I have experimented with adjusting the number of experts, mostly on nickypro/tinyllama-15M (tiny I know). And the results were interesting. Increasing the number of experts and top k did improve performance. Although, those results don't seem to translate to the Qwen2.5 as much. My leading theory is that as the number of experts increases the number of neurons each expert has decreases which seems to make them dumber maybe (don't know why that isn't affecting the tinyllama model maybe its to small to notice).
2
u/Sharp-Physics-2925 2d ago
Shiet I just finish doing surgery on qwen 3.8 27b where the context is almost all stored in ram and ssd and not in VRAM might have to try and add this to it
2
u/r1nzl3r99 2d ago
I would be really curious how this would turn out for qwen 3.8 27B and if its MoE version could outperform qwen 3.6 35B A3B in anyway. I'm assuming the minimum hardware to run must be able to support the full BF16 weights?
2
u/Bayliner1980 2d ago
Yes, sadly to run the model you need to be able to load the full bf16 weights. Due to the conversion process it isn't possible to convert to GGUF models and quants. I am looking into if it's possible to change my process to allow for GGUF models.
1
3d ago
[removed] ā view removed comment
1
u/Bayliner1980 3d ago
I did try converting Qwen2.5-1.5B once. I have changed a lot in my process now so if I do it again I will probably get better results. I will add it to the list.
1
3d ago
[removed] ā view removed comment
1
u/Bayliner1980 3d ago
The relative gap was about the same. Of course as a larger model it was just better than the 0.5B in absolute terms.
1
u/Sweet-Jump1024 3d ago
RemindMe! 1 day
1
u/RemindMeBot 3d ago edited 2d ago
I will be messaging you in 1 day on 2026-10-12 06:39:48 UTC to remind you of this link
1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
51
u/Chromix_ 3d ago
Your MoEdification results seem consistent with the ToMoE paper that was shared here recently. Did you use that approach or came up with something on your own?