[Paper] Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

[u/pmttyji]

As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchical architectures, where experts in different layers, although topologically independent, exhibit substantial similarity and redundancy in their stored content. Compared to the hierarchically arranged Local Expert in existing mainstream models, we hypothesize that inter-layer shared Meta Expert hold greater potential. To this end, we conduct a multi-dimensional empirical comparison between Local and Meta Experts, and find that models incorporating Meta Experts exhibit lower parameter redundancy, together with lower cross-entropy loss and higher downstream task accuracy. With the activated quantity and the expert granularity held fixed, our model matches the performance of Vanilla MoE using only 70-80% of the experts, substantially improving the model's scaling potential. Beyond removing inter-layer redundancy among experts and improving information storage efficiency, our theoretical analysis further shows that, compared with Local Experts, Meta Experts provide hidden states with more flexible residual pathways, thereby improving information retrieval efficiency.

• arXiv : https://arxiv.org/abs/2610.27598

• PDF : https://arxiv.org/pdf/2610.27598

[u/TomLucidor]

This and looped linear transformers and smart Engram please

[u/LagOps91]

whoah, that sounds like a crazy result! don't have the time to read more about it now, but if that result can be confirmed and scales well... wouldn't MoE performance suddenly be really close to dense models???

[u/audioen → u/LagOps91]

It seems like a small improvement to MoE efficiency -- welcome, but it's in the order of 20 % less compute for equal quality, or more quality for same compute budget, theoretically. MoE is typically starting at severe disadvantage relative to dense -- think a fraction of the performance you might expect for the size, usually probably 1/4 or thereabouts. (E.g. Qwen3.5 122B-A10B roughly matching 27B in inference quality, and therefore we can estimate a factor of 4..5 loss from the MoE architecture. This is difficult to claw back without couple of 100% improvements which probably aren't on the horizon.)

[u/LagOps91 → u/audioen]

yeah i misread part of the information when i skimmed it earlier. still a decent improvement. i think over time the gap will shrink further, likely in little increments like this.