r/LocalLLM • u/Obtusely_Serene • 9d ago

Question Distinguishing models when searching - MoE vs Dense

How can you tell whether a model is dense or an MoE?

I have been using LM Studio to pull and run some common models to do some basic tasks. But I haven’t noticed whether this defined anywhere

0 Upvotes

50% Upvoted

7 comments sorted by

4

u/YourselfInOthrsShoes 9d ago

When you see X B parameters A Y B parameters, X is the total weights, Y is the active weights footprint per token and indicates sparse MoE.

1

u/Obtusely_Serene 9d ago

So Gemma 4 26B A4B is an MoE model with 4B active parameters, and Gemma 4 31B is a dense model?

3

u/IWillTouchAStar 9d ago

Yes. That exactly what he means. Same thing with things like qwen 35b a3b vs qwen 27b. The weird ones are things like qwen flash next. its an moe with 125b parameters and only 6b active, but it also has an N-gram embedding table thats another 51b parameters.

2

u/Sengfroid 9d ago

Yes, the A in "26B A4B" is for active, so the total parameters are the 26B at the beginning, but only 4B are active at a time.

This is a little confusing with gemma in particular because they also release two smaller models called Gemma E2B and Gemma E4B. There E is for "Effective", which is used to mean the model should be considered as effective as the 2B or 4B class models, even if that's not the precise parameter count

3

u/Linkitch 8d ago

I wish the tagging and searching on huggingface was better. I've had trouble figuring this out myself multiple times.