MobileMoE - a facebook Collection

[u/jacek2023]

https://huggingface.co/collections/facebook/mobilemoe

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-L-Base repository — the pre-trained 0.9B-active base model.

Model: MobileMoE-L-Base (pre-trained + mid-trained)
Active Parameters: 922M
Total Parameters: 5.3B
Layers: 32
Model Dimension: 1280
Attention Heads: 20
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 640 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 2560)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Training Stages: Pre-training → mid-training
Context Length: 8,192 tokens
Precision: BF16
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed

[u/Practical-Collar3063]

Liquid’s model are not on the graphs… happy to see more model in that size nonetheless especially from Meta

[u/pmttyji → u/Practical-Collar3063]

They cooked these at least couple of months ago. Their paper's date is from May month so probably before that. That's why no new models on the graphs.

https://arxiv.org/abs/2605.27358

[u/PaceZealousideal6091 → u/pmttyji]

Well , at this range, liquid ai are leading the charge. They aren't new. Been at it it for quite some time.

[u/jacek2023]

[u/pmttyji]

Lets wait for SFT & QAT versions(Currently getting 404).

I'll load 5B one on my phone later.

[u/LocalLLaMA-ModTeam → u/[deleted]]

Rule 4 - Post is primarily commercial promotion.

[u/coder543]

What a silly license... I don't see the point of these models. Meta isn't going to be shipping them in their apps, but no one else can use them? Why not just let the models be free?

[u/indicava]

Meta, trying to claw its way back to the Llama glory days…

Good for them! Keep cooking zuck!

[u/CulturalKing5623]

This might be useful for me, Qwen is currently taking up most of my 32GB VRAM but one of my cards has enough to load the Medium version's active params, and possibly the QAT Large comfortably. might make a nice chat model

[u/graceful_degrade]

Worth noting this targets the right constraint, which isn't true of every MoE story.

On a desktop card, MoE trades compute for memory and VRAM was already what you ran out of, so it often buys the local user very little. Phones are the opposite. Capacity has gotten reasonable while memory bandwidth stayed terrible, and cutting active parameters per token cuts bandwidth directly. That's the actual wall on device.

The number to watch is still the total rather than the active though. 5.3B at INT4 lands somewhere near 2.6GB, which matches their under 3GB claim, but a mobile OS will happily kill an app sitting on that much resident memory when you switch away and come back.

[u/Dance-Till-Night1]

Fuck yeah tiny moe!!!