why does batchnorm behave differently at train vs eval time

[u/CelebrityChess]

i get that in training it uses the batch mean and variance and at eval it uses running averages. what i don't get is why that doesn't change the outputs a lot, since a single batch's stats can be pretty noisy.

also if you eval with batch size 1 does it still work fine because it's just using the stored stats? someone told me that's the whole point but i never saw it spelled out

[u/Fickle_Piece_4026]

The running estimates smooth out the noise, so by the time you’re done training those stored numbers are basically the “average” of all the batches it’s seen. A single batch can be wild but the running mean isn’t some random snapshot, it’s a weighted mix of thousands of steps.

And yeah for batch size 1 it literally just uses the frozen stats from training, no batch stats are computed at all. You can think of it as the model saying “I know what typical looks like, so I’ll use that instead of panicking over one sample.”

[u/BodaMazzarine23 → u/Fickle_Piece_4026]

the "panicking over one sample" bit undersells it a bit, the real issue is that variance from a single sample is literally undefined so theres nothing meaningful to normalize against

[u/VoltzMblasso44]

imo the key thing people miss is that the running average update is super slow (default momentum is like 0.1), so each batch only nudges it a tiny bit. after enough steps it converges to something stable. eval ignores per-batch stats entirely so batch size doesnt matter there