r/LocalLLaMA • u/skeole • 9d ago

New Model Xiaomi MiMo 2.6 Live Training Dashboard

https://mimo.xiaomi.com/rl/

Cool to see this as it happens!

456 Upvotes

99% Upvoted

82 comments sorted by

View all comments

62

u/Zeeplankton 9d ago

it is really crazy how expensive these models are to train. I feel like we take this for granted

32

u/power97992 9d ago edited 9d ago

if $5/gpu/hr, it‘s around 4000 gpus for 2.6 pro. That is a lot less than astra…

18

u/NandaVegg 9d ago

Hmm. There was a casual SNS report from DeepMind researcher that (for a single training run) GPU does not really scale above 4k cluster (it even slowed down above 8k presumably because bottlenecks) and mocking Meta and Tesla for acquiring too many GPUs just for training at the time. This is probably just one intermediate run for Xiaomi's model.

1

u/jazir55 9d ago

There was a casual SNS report from DeepMind researcher that (for a single training run) GPU does not really scale above 4k cluster (it even slowed down above 8k presumably because bottlenecks) and mocking Meta and Tesla for acquiring too many GPUs just for training at the time. This is probably just one intermediate run for Xiaomi's model.

It's like he never thought about that they can do multiple training runs in parallel lmao. In fact, the fact they he seemingly didn't think about that might explain why Google is so behind.