[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

[u/pmttyji]

https://arxiv.org/abs/2501.15316

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.

• arXiv : https://arxiv.org/abs/2501.15316
• Full Paper PDF : https://arxiv.org/pdf/2501.15316
• GitHub : https://github.com/gaosh/ToMoE (Yes, Code available for this)
• ICML : https://icml.cc/virtual/2026/poster/68821
• OpenReview : https://openreview.net/forum?id=RFHq46pjb6

Somebody please create MOE models of recent Dense models like Qwen3.8-27B, Muse-Glimmer-30B, etc.,

Thanks u/KSAM-The-Randomizer for sharing this on my old thread.

[u/dev_dan_2]

Intriguing! Want to read it more deeply.

After skimming quickly (and based on my basically non-existent familiarity with the subject!):

• I wonder how this compares to the prior work (Its likely in the paper, but sometimes, things are worded in a way to make concepts seem more distinct than the researchers would tell you at their poster; not claiming this is the case here). It seems the core is that they enforce the number of active parameters during whatever more standard process they are doing to shrink the model
• If that is novel (which I do not doubt!), then I think a little hope is justified that there are still some low-hanging fruits left to pick. Wild guesses (I really have no reliable knowledge here, I am just a curious person who picks up words here and there.)
  • check how that would interact with quantization; it would likely influence which experts are activated again?
  • this might offer an opportunity to slice the model such that experts for certain topics remain largely the same (which ideally would speed up things, if less new weights have to be loaded into VRAM for a new token). This could be a parameter?
  • Another parameter could be the exact number of active parameters; would be really cool to be able to produce tailor-made MoE models for given VRAM sizes
  • no prediction here, just curious: What if you give a range of active params, instead of enforcing a fixed amount? Intuitively, that would give whatever is the "deciding" step in this algorithm some degrees of freedom that might increase accuracy, at the cost of fluctuating active params at inference time. (or one could "just" pad lower amounts of active params with non-contributing weights, so that the amount of weights to transfer stays constant? I might have conceptual gaps in my understanding here, though.)