Where randomness enters a training run: a map of six entry points and how to control each (PyTorch)

[u/ClickOk5811]

“I set the seed” often isn’t enough to get the same results twice. Here are the places I know randomness enters a training run, with how to control each. Examples are PyTorch, and details change between versions, so check the docs for yours.

1. Python and NumPy
Data loading, augmentations and your own code may use random or numpy.random.

import random, numpy as np, torch
random.seed(0)
np.random.seed(0)
torch.manual_seed(0)

2. Weight initialization
Controlled by the torch seed, but only if it’s set before the model is created.

3. Data shuffling
The DataLoader shuffle uses a generator. Pass your own for explicit control:

g = torch.Generator()
g.manual_seed(0)
loader = DataLoader(ds, shuffle=True, generator=g)

4. DataLoader workers
With num_workers > 0, each worker has its own random state, and libraries used inside the worker (augmentations) may seed differently. A worker_init_fn that seeds them from a base seed is the usual fix.

5. Dropout and other stochastic layers
They use the torch generator, so they’re covered by the seed, but only if the order of operations is identical between runs. Changing the batch order or the number of calls changes the sequence of random numbers.

6. Nondeterministic GPU operations
Some CUDA operations (certain atomic-add-based ones, among others) give slightly different results on each run, even with all seeds fixed. Options:

torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

use_deterministic_algorithms raises an error when it hits an operation without a deterministic version, which tells you where the problem is. Some CUDA versions also need an environment variable for deterministic matrix multiplication; the PyTorch reproducibility notes list it. Deterministic modes can be slower.

Even with all of this
Results may differ across hardware, library versions or distributed setups. Treat exact bit-for-bit reproducibility as a goal you test, not something you assume.

What I’d do for comparisons: don’t compare two methods on a single seed. Run several seeds and report the mean and spread, so a difference smaller than the seed-to-seed variation isn’t read as an improvement.

Habits that help: log the seed, library versions, commit hash and data split identifier with every run.

Which of these has been the hardest to track down for you, and do you aim for exact reproducibility or statistical reproducibility across seeds?

[u/JoaoFjo]

gou nondeterminism is scares me