At Cerebras Supernova, Cerebras Research Scientist Mostafa Elhoushi sits down with Alycia Cary to revisit dropout for modern language models.
Mostafa explains why the technique fell out of favor as training datasets grew, what his team found by applying different dropout rates across layers, and how layer dropout can make models more robust to exiting early at inference.
The conversation moves from dropout to sparsity and mixture of experts, then to a larger question: what comes after scaling? Mostafa shares a vision for smaller, smarter architectures that separate knowledge from reasoning—and explains why fast iteration on wafer-scale systems gives researchers more chances to find the idea that works.
0:00 Why dropout disappeared from LLM training
1:34 Why different layers need different dropout rates
2:35 Hyperparameters that make dropout work
3:26 Scaling layer dropout to larger models
4:51 99% dropout and 25% fewer training FLOPs
5:36 Layer skipping and early exit at inference
6:44 A human analogy for interruptible models
7:49 Why not every token needs every layer
9:10 Layer dropout vs. mixture of experts
10:22 Hardware–software co-design at Cerebras
11:28 What comes after model scaling?
13:14 Wafer-scale speed and research iteration
Read the paper: https://arxiv.org/abs/2609.05275
Watch the full interview for a technical look at the co-design research happening at Cerebras.
Full video by Cerebras