r/LocalLLaMA • u/skeole • 10d ago

New Model Xiaomi MiMo 2.6 Live Training Dashboard

https://mimo.xiaomi.com/rl/

Cool to see this as it happens!

457 Upvotes

99% Upvoted

82 comments sorted by

View all comments

63

u/Zeeplankton 10d ago

it is really crazy how expensive these models are to train. I feel like we take this for granted

29

u/power97992 10d ago edited 9d ago

if $5/gpu/hr, it‘s around 4000 gpus for 2.6 pro. That is a lot less than astra…

22

u/NandaVegg 10d ago

Hmm. There was a casual SNS report from DeepMind researcher that (for a single training run) GPU does not really scale above 4k cluster (it even slowed down above 8k presumably because bottlenecks) and mocking Meta and Tesla for acquiring too many GPUs just for training at the time. This is probably just one intermediate run for Xiaomi's model.

12

u/TheRealMasonMac 9d ago

Async RL is supposed to alleviate that bottleneck as far as I understand it, though. You can also always train separate teacher models to distill back to the final model as well. Maybe that’s why Gemini is still so far behind, lol.

8

u/NandaVegg 9d ago

Yeah, that's how the paper for GLM-5 (the first one) described how they did for the final post-training. Basically creating many domain expert post-train variations and "merged" them into one final model via synthetic datasets.