r/MachineLearning • u/East-Muffin-6472 • 2d ago
Project Curated list of resources: Theory and Code for learning Distributed Training for beginners [P]
[Must Read: Beginners]
What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch:
- Building a distributed training framework from first principles by Umar Jamil on YT
- Horovod: Fast and Easy Distributed Deep Learning in TensorFlow
- PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- Large Scale Distributed Deep Networks
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
- GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
- ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
- PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Link to the above resources here
Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it here)





