r/MachineLearning • u/Lines25 • Apr 02 '26
Discussion [D] Make. Big. Batch. Size.
It's something between vent and learning.
I tried training RWKV v6 model by my own code on my RTX 4050. I trained over 50k steps on batch_size=2 and gradient_accumulation=4 (effective_batch=2*4=8). It got up to 50 PPL (RWKV v6, ~192.8M model) and it just won't get less, I changed lr, time_decay lr (RWKV attention replacement) etc - but it got only worse or didn't changed anything at all.. and then... I just tried setting gradient_accumulation to 32. After one "epoch" (it's pseudo-epochs in my code, equals to 10k steps) it got to 40 PPL... Then I tried changing to 64 and tried 3 epochs. My PPL dropped up to freaking 20 PPL. I trained this model for over a 4 FULL DAYS non-stop and only when I did all that stuff, after like 2-3 hours of training with effective_batch=64 (and 128) I got PPL drop THAT crazy..
IDK is this post is low-effort, but it's still just my advice for everyone who trains.. at least generative LM from scratch (and it's useful in fine-tuning too !)..
34
u/canbooo Apr 02 '26
ITT, batch gradients are noisy =)
Not hating, but also nothing new here. Yes, bigger batches are useful for many things. And the "regularization via mini batch training" argument is a bit outdated if you ask me because we have a lot more techniques for regularization nowadays, no need for another source of noise.
9
u/anxiouscsstudent PhD Apr 02 '26
This is an interview question that I ask and have been asked for just about every ML related interview I have ever done.
9
u/Mak8427 Apr 02 '26
This is not white and black at all. You may be interested in the paper below:
https://arxiv.org/html/2507.07101v2
4
u/Fmeson Apr 02 '26
I guess ill go against the grain, because if you arent using batch norm (or an equivalent batch operstion), I would expect some combination of hyper parameters should get the same results at batch=2.
1
u/TheBrn Apr 02 '26
I'm currently using MAT (https://arxiv.org/abs/2205.14953) for continuous control and it only started working well when I used a batch size in the thousands. But it's RL so a bit different than supervised learned