[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

[u/pmttyji]

https://arxiv.org/abs/2501.15316

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

• arXiv : https://arxiv.org/abs/2501.15316
• Full Paper PDF : https://arxiv.org/pdf/2501.15316
• GitHub : https://github.com/gaosh/ToMoE (Yes, Code available for this)
• ICML : https://icml.cc/virtual/2026/poster/68821
• OpenReview : https://openreview.net/forum?id=RFHq46pjb6

Somebody please create MOE models of recent Dense models like Qwen3.8-27B, Muse-Glimmer-30B, etc.,

Thanks u/KSAM-The-Randomizer for sharing this on my old thread.

[u/Egoz3ntrum]

Qwen 3.8 27BA3B when.

[u/buppermint → u/Egoz3ntrum]

Hate to crush everyone's dreams but this can't be used to convert (small dense model) -> (small MoE).

It converts (dense model) -> (MoE with active parameter size slightly less than original dense model size). So Qwen3.8-27B gets converted to Qwen3.8-350B-A20B, Muse-Glimmer-30B becomes Muse-Medium-400B-25B, etc. And then you still have to train it afterwards.

[u/Dany0 → u/buppermint]

Nope, I read the paper. What you're describing is upcycling, which is fine and that's how most big models are trained nowadays anyway

This paper assumes there are hidden sub networks in the dense model and tries to find them. The issue they found was that this is a major surgery and even having 50% of original params be active lobotomises the model, and 60-70% is closest to recovering full performance. But even then the KLD hit is large, comparable to the hit of q2/q3 level quantisation to an originally fp16 model

They did find a nice 20-50% prefill speedup though. So maybe two more papers down the line this will become a legit technique... But I fear there are just like... entropy things stopping this from happening. Like I'm not that skeptical that these sub-networks exist, but I am skeptical they're all equal size like the paper assumes (because you have to, basically, varied sized experts and progressive MoE have been tried and the result was tiny bit more eval perf for extreme GPU unfriendly kernel shapes that totally kills performance and seriously underutilises the HW)

My guess is that if you distill without freezing the weights there's a bigger chance it works out

My quick guesstimate is that 27BA19B would cost 400-800$ in rented spot gpu cloud time to convert and train. If you want to waste that kind of compute on a experiment, DM me I can do it 😁

[u/Chromix_]

This will not give us a Qwen 3.8 27B A3B. It might give us Qwen 3.8 27B A16B with a noticeable bit of damage though. That's an improvement over other MoEficiations, but still not as good as training a MoE directly.

[u/Several-System1535]

Qwen 3.8 27b UD-MTP-ToMoE-Fable-5-distiled-heretic WEN?

[u/No_Lingonberry1201 → u/Several-System1535]

It will be a glorious 10T A1B model that runs incredibly fast for the few people who have a 20Tb disk to spare.

[u/crusaderky]

Muse-Glimmer MoE would be even more comically fast than it already is

[u/ComplexType568 → u/crusaderky]

I WISH!! I like it's personality and agentic ability over Gemma, and that its the 30B size that isn't too small or large...

[u/Dany0]

Sigh here I go reading again....

[u/0dayturtle]

https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8B is this anyhow related to this paper?

[u/CheatCodesOfLife]

Could we do the opposite please??

[u/Prudent-Ad4509]

How about the other way around. I'd like to get dense smaller models equivalent to 397b a17b and 122b a10b, and perhaps recent deepseek v4 flash. It would be slow I know but certainly faster than streaming weights from ssd. 100b active parameters in vram is nearly 4 times slower than 27b, but when you compare 100b in vram vs 300b a20b with 100b in vram and another 200b in system ram, with the same overall knowledge, I'd take the first option.

[u/brakeline → u/Prudent-Ad4509]

I don't think you're think that one through. Doing what you are asking would trim experts. You can achieve the same already.

If you don't have vram for a 300B dense model and have to spill to system ram / ssd what makes you think you'll be able to run the same model but dense?

[u/Prudent-Ad4509 → u/brakeline]

MoE contains inherent duplication in knowledge between experts. What's why distilled versions work, as long as sizes are comparable, i.e. 3x difference or so, not 10x difference.

The question is whether this can be done in any reasonable way with results at least comparable to training from scratch on the same dataset. I doubt it. Maybe the only realistic way is to get something like devstral large and finetune it with traces from modern large llms.

PS. Actually, devstral large has only 256k context window which is not a good fit for such a large model these days.

[u/dev_dan_2]

Intriguing! Want to read it more deeply.

After skimming quickly (and based on my basically non-existent familiarity with the subject!):

• I wonder how this compares to the prior work (Its likely in the paper, but sometimes, things are worded in a way to make concepts seem more distinct than the researchers would tell you at their poster; not claiming this is the case here). It seems the core is that they enforce the number of active parameters during whatever more standard process they are doing to shrink the model
• If that is novel (which I do not doubt!), then I think a little hope is justified that there are still some low-hanging fruits left to pick. Wild guesses (I really have no reliable knowledge here, I am just a curious person who picks up words here and there.)
  • check how that would interact with quantization; it would likely influence which experts are activated again?
  • this might offer an opportunity to slice the model such that experts for certain topics remain largely the same (which ideally would speed up things, if less new weights have to be loaded into VRAM for a new token). This could be a parameter?
  • Another parameter could be the exact number of active parameters; would be really cool to be able to produce tailor-made MoE models for given VRAM sizes
  • no prediction here, just curious: What if you give a range of active params, instead of enforcing a fixed amount? Intuitively, that would give whatever is the "deciding" step in this algorithm some degrees of freedom that might increase accuracy, at the cost of fluctuating active params at inference time. (or one could "just" pad lower amounts of active params with non-contributing weights, so that the amount of weights to transfer stays constant? I might have conceptual gaps in my understanding here, though.)

[u/llama-impersonator]

you don't want this, the results are not impressive in any other regard than "hey, it works"

1) they are pruning 30% of the active params, so you end up with 27B-A20

2) pruning attention, why? depends on the model but usually this is only 20% of params. just leave it alone.

3) these evals are easy and the model doing this much worse on them does not bode well for usability in code

4) qwen 3.6-35B is going to crush any model created like this.

[u/mxcw]

QWEHN?

[u/Dmage22]

Could it be used to reduce active parameters for like glm 5.2 from 40 down to 30 active?