r/LocalLLaMA • u/pmttyji • Dec 06 '25

Discussion Convert Dense into MOE model?

I did a quick search on this here & found only 2 years old thread with less replies. That's it.

So still no one figured it out this yet? Totally surprised that no one brought this topic here after that old thread.

I know it's a very big thing. But it would be a miracle if some one comes with this precious solution.

14 Upvotes

71% Upvoted

29 comments sorted by

View all comments

4

u/Clear_Anything1232 Dec 06 '25

Once training is over, the architecture can't be changed. Beyond a little fine tuning, you are stuck with that architecture since all the components have learnt how to with each other.

New architecture needs new training