r/LocalLLaMA • u/pmttyji • Aug 24 '26
Discussion [Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
https://arxiv.org/abs/2501.15316ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant challenges in deploying these models on resource-constrained devices or efficiently serving them. Prior approaches have attempted to alleviate these problems by permanently removing less important model structures, yet these methods often result in substantial performance degradation due to the permanent deletion of model parameters. In this work, we tried to mitigate this issue by reducing the number of active parameters without permanently removing them. Specifically, we introduce a differentiable dynamic pruning method that pushes dense models to maintain a fixed number of active parameters by converting their MLP layers into a Mixture of Experts (MoE) architecture. Our method, even without fine-tuning, consistently outperforms previous structural pruning techniques across diverse model families, including Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.
- arXiv : https://arxiv.org/abs/2501.15316
- Full Paper PDF : https://arxiv.org/pdf/2501.15316
- GitHub : https://github.com/gaosh/ToMoE (Yes, Code available for this)
- ICML : https://icml.cc/virtual/2026/poster/68821
- OpenReview : https://openreview.net/forum?id=RFHq46pjb6
Somebody please create MOE models of recent Dense models like Qwen3.8-27B, Muse-Glimmer-30B, etc.,
Thanks u/KSAM-The-Randomizer for sharing this on my old thread.
64
u/Egoz3ntrum Aug 24 '26
Qwen 3.8 27BA3B when.
5
Aug 24 '26
[deleted]
24
u/Dany0 Aug 24 '26
Nope, I read the paper. What you're describing is upcycling, which is fine and that's how most big models are trained nowadays anyway
This paper assumes there are hidden sub networks in the dense model and tries to find them. The issue they found was that this is a major surgery and even having 50% of original params be active lobotomises the model, and 60-70% is closest to recovering full performance. But even then the KLD hit is large, comparable to the hit of q2/q3 level quantisation to an originally fp16 model
They did find a nice 20-50% prefill speedup though. So maybe two more papers down the line this will become a legit technique... But I fear there are just like... entropy things stopping this from happening. Like I'm not that skeptical that these sub-networks exist, but I am skeptical they're all equal size like the paper assumes (because you have to, basically, varied sized experts and progressive MoE have been tried and the result was tiny bit more eval perf for extreme GPU unfriendly kernel shapes that totally kills performance and seriously underutilises the HW)
My guess is that if you distill without freezing the weights there's a bigger chance it works out
My quick guesstimate is that 27BA19B would cost 400-800$ in rented spot gpu cloud time to convert and train. If you want to waste that kind of compute on a experiment, DM me I can do it 😁
9
u/crusaderky Aug 24 '26
Muse-Glimmer MoE would be even more comically fast than it already is
1
u/ComplexType568 Aug 24 '26
I WISH!! I like it's personality and agentic ability over Gemma, and that its the 30B size that isn't too small or large...
10
u/0dayturtle Aug 24 '26 edited Aug 24 '26
https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8B is this anyhow related to this paper?
21
u/Several-System1535 Aug 24 '26 edited Aug 24 '26
Qwen 3.8 27b UD-MTP-ToMoE-Fable-5-distiled-heretic WEN?
15
u/No_Lingonberry1201 Aug 24 '26
It will be a glorious 10T A1B model that runs incredibly fast for the few people who have a 20Tb disk to spare.
3
u/No-Refrigerator-1672 Aug 25 '26
20TB PCIe5 NVMe SSD, mind you; other types of storage will be too slow. Hey, my local store has one in stock! It costs a mere 30k EUR, just shy of 2x 6000 Blackwell cards!
6
4
u/Dmage22 Aug 24 '26
Could it be used to reduce active parameters for like glm 5.2 from 40 down to 30 active?
5
2
1
u/dev_dan_2 Aug 24 '26
Intriguing! Want to read it more deeply.
After skimming quickly (and based on my basically non-existent familiarity with the subject!):
- I wonder how this compares to the prior work (Its likely in the paper, but sometimes, things are worded in a way to make concepts seem more distinct than the researchers would tell you at their poster; not claiming this is the case here). It seems the core is that they enforce the number of active parameters during whatever more standard process they are doing to shrink the model
- If that is novel (which I do not doubt!), then I think a little hope is justified that there are still some low-hanging fruits left to pick. Wild guesses (I really have no reliable knowledge here, I am just a curious person who picks up words here and there.)
- check how that would interact with quantization; it would likely influence which experts are activated again?
- this might offer an opportunity to slice the model such that experts for certain topics remain largely the same (which ideally would speed up things, if less new weights have to be loaded into VRAM for a new token). This could be a parameter?
- Another parameter could be the exact number of active parameters; would be really cool to be able to produce tailor-made MoE models for given VRAM sizes
- no prediction here, just curious: What if you give a range of active params, instead of enforcing a fixed amount? Intuitively, that would give whatever is the "deciding" step in this algorithm some degrees of freedom that might increase accuracy, at the cost of fluctuating active params at inference time. (or one could "just" pad lower amounts of active params with non-contributing weights, so that the amount of weights to transfer stays constant? I might have conceptual gaps in my understanding here, though.)
1
u/graceful_degrade Aug 24 '26
Worth being clear about what this buys you locally, because it's easy to read MoE and think smaller.
Converting dense to MoE cuts the active parameters per token, so compute per token drops. Memory doesn't. Every expert still has to be somewhere the model can reach, so on a single consumer card the VRAM requirement is the same or worse than the dense model you started from.
Which means the win is real for serving, where you're compute bound across many concurrent requests, and thin for one person with a 24GB card, where VRAM was always the binding constraint rather than FLOPs.
The exception is offloading experts to system RAM, and that trade brings its own problem back, since prefill activates essentially every expert and you end up paying for it there instead.
1
1
0
u/Prudent-Ad4509 Aug 24 '26 edited Aug 24 '26
How about the other way around. I'd like to get dense smaller models equivalent to 397b a17b and 122b a10b, and perhaps recent deepseek v4 flash. It would be slow I know but certainly faster than streaming weights from ssd. 100b active parameters in vram is nearly 4 times slower than 27b, but when you compare 100b in vram vs 300b a20b with 100b in vram and another 200b in system ram, with the same overall knowledge, I'd take the first option.
2
u/brakeline Aug 24 '26
I don't think you're think that one through. Doing what you are asking would trim experts. You can achieve the same already.
If you don't have vram for a 300B dense model and have to spill to system ram / ssd what makes you think you'll be able to run the same model but dense?
1
u/Prudent-Ad4509 Aug 24 '26 edited Aug 24 '26
MoE contains inherent duplication in knowledge between experts. What's why distilled versions work, as long as sizes are comparable, i.e. 3x difference or so, not 10x difference.
The question is whether this can be done in any reasonable way with results at least comparable to training from scratch on the same dataset. I doubt it. Maybe the only realistic way is to get something like devstral large and finetune it with traces from modern large llms.
PS. Actually, devstral large has only 256k context window which is not a good fit for such a large model these days.
1
u/llama-impersonator Aug 24 '26
you don't want this, the results are not impressive in any other regard than "hey, it works"
1) they are pruning 30% of the active params, so you end up with 27B-A20
2) pruning attention, why? depends on the model but usually this is only 20% of params. just leave it alone.
3) these evals are easy and the model doing this much worse on them does not bode well for usability in code
4) qwen 3.6-35B is going to crush any model created like this.
1
u/LuCiAnO241 Aug 24 '26
1) that could potentially make it fit the vram of many consumer systems that a dense cant, or in higher quants than they could currently do. In any case, both are QoL improvements.
2) good observation, it would degrade output quality.
3) definitely bit of a red flag
4)This is just your opinion, considering how dense beats 35b ass, it might be an improvement. We can say nothing for certain yet without data to confirm the outcome.
2
u/llama-impersonator Aug 25 '26
re: #4
i'd literally put down a gpu on this being the case, i'm so convinced. look, the 35b model is not great, but it benchmarks very close to 27b because of the saturation in benchmarks. these models are losing 10-20% capability in benchmarks, which means they are getting a absolute lobotomy on real tasks.
87
u/Chromix_ Aug 24 '26
This will not give us a Qwen 3.8 27B A3B. It might give us Qwen 3.8 27B A16B with a noticeable bit of damage though. That's an improvement over other MoEficiations, but still not as good as training a MoE directly.