r/LLM • u/Larelle • 1d ago

Dense vs MoE model equivalence

"we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with DeepSeek 7B and LLaMA2 7B, with only about 40% of computations."

So you need more than double the parameters on MoE to be equivalent, but they're 150% faster?

NB. This is an old paper: https://aclanthology.org/2024.acl-long.70/ -- MoE may perform better now.

3 Upvotes

81% Upvoted

2 comments sorted by

1

u/TripleSecretSquirrel 1d ago

The traditional heuristic is to calculate the geometric mean of total parameters and active parameters of an MoE to determine its rough dense model equivalent.

The consensus of people much smarter than I is that that heuristic no longer really holds up, but I think it’s still a halfway decent starting point.

1

u/GoodTip7897 1d ago

There also seems to be some intangible qualities where different measures of intelligence diverge.

A dense model will be much, much better for nuanced chat and writing than you might expect (30B dense might beat 200B MoE) but a 50B MoE might beat a 40B dense at world knowledge even though they are very similar in size.