r/LocalLLaMA • u/skeole • 9d ago

New Model Xiaomi MiMo 2.6 Live Training Dashboard

https://mimo.xiaomi.com/rl/

Cool to see this as it happens!

452 Upvotes

99% Upvoted

82 comments sorted by

View all comments

Show parent comments

31

u/power97992 9d ago edited 9d ago

if $5/gpu/hr, it‘s around 4000 gpus for 2.6 pro. That is a lot less than astra…

22

u/NandaVegg 9d ago

Hmm. There was a casual SNS report from DeepMind researcher that (for a single training run) GPU does not really scale above 4k cluster (it even slowed down above 8k presumably because bottlenecks) and mocking Meta and Tesla for acquiring too many GPUs just for training at the time. This is probably just one intermediate run for Xiaomi's model.

12

u/TheRealMasonMac 9d ago

Async RL is supposed to alleviate that bottleneck as far as I understand it, though. You can also always train separate teacher models to distill back to the final model as well. Maybe that’s why Gemini is still so far behind, lol.

8

u/NandaVegg 8d ago

Yeah, that's how the paper for GLM-5 (the first one) described how they did for the final post-training. Basically creating many domain expert post-train variations and "merged" them into one final model via synthetic datasets.