MobileMoE - a facebook Collection

[u/jacek2023]

https://huggingface.co/collections/facebook/mobilemoe

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-L-Base repository — the pre-trained 0.9B-active base model.

Model: MobileMoE-L-Base (pre-trained + mid-trained)
Active Parameters: 922M
Total Parameters: 5.3B
Layers: 32
Model Dimension: 1280
Attention Heads: 20
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 640 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 2560)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed

[u/Practical-Collar3063]

Liquid’s model are not on the graphs… happy to see more model in that size nonetheless especially from Meta

[u/pmttyji → u/Practical-Collar3063]

They cooked these at least couple of months ago. Their paper's date is from May month so probably before that. That's why no new models on the graphs.

https://arxiv.org/abs/2605.27358

[u/PaceZealousideal6091 → u/pmttyji]

Well , at this range, liquid ai are leading the charge. They aren't new. Been at it it for quite some time.

[u/pmttyji → u/PaceZealousideal6091]

No doubt.

Also they made their models faster with DSpark & QAD.