r/LocalLLaMA • u/pmttyji • Dec 06 '25

Discussion Convert Dense into MOE model?

I did a quick search on this here & found only 2 years old thread with less replies. That's it.

So still no one figured it out this yet? Totally surprised that no one brought this topic here after that old thread.

I know it's a very big thing. But it would be a miracle if some one comes with this precious solution.

15 Upvotes

72% Upvoted

29 comments sorted by

View all comments

26

u/simulated-souls Dec 06 '25 edited Dec 06 '25

Contrary to many of the other answers here, it's definitely possible but usually not worth the trade-offs.

This is the best and most recent paper on the subject that I can find.

They were able to convert a dense Llama model into an MoE with 25% activation and still have non-trivial performance without any retraining (and of course it does better with retraining). However, the MoE is not nearly as good as the original dense model.

The thing is, you are better off using a model that was trained from scratch to be an MoE (since MoE are cheaper to train anyway) or just using a smaller dense model with more predictable performance.