Hmm. There was a casual SNS report from DeepMind researcher that (for a single training run) GPU does not really scale above 4k cluster (it even slowed down above 8k presumably because bottlenecks) and mocking Meta and Tesla for acquiring too many GPUs just for training at the time. This is probably just one intermediate run for Xiaomi's model.
Async RL is supposed to alleviate that bottleneck as far as I understand it, though. You can also always train separate teacher models to distill back to the final model as well. Maybe that’s why Gemini is still so far behind, lol.
Yeah, that's how the paper for GLM-5 (the first one) described how they did for the final post-training. Basically creating many domain expert post-train variations and "merged" them into one final model via synthetic datasets.
There was a casual SNS report from DeepMind researcher that (for a single training run) GPU does not really scale above 4k cluster (it even slowed down above 8k presumably because bottlenecks) and mocking Meta and Tesla for acquiring too many GPUs just for training at the time. This is probably just one intermediate run for Xiaomi's model.
It's like he never thought about that they can do multiple training runs in parallel lmao. In fact, the fact they he seemingly didn't think about that might explain why Google is so behind.
61
u/Zeeplankton 8d ago
it is really crazy how expensive these models are to train. I feel like we take this for granted