Reproducibility

Problem

A training run is full of random choices: initial weights, the order of the data, dropout masks, augmentations, which examples land in the validation split. Change any of them and the final model changes, sometimes by more than the effect you are trying to measure. If two runs of the same script give accuracies of 91.2% and 90.6%, you cannot tell whether your new idea helped by 0.3 points, and you cannot debug a loss spike you cannot make happen again.

Fixing a seed is the start, not the end. Data-loader workers run in separate processes with their own generators. Some GPU kernels add numbers in whatever order threads finish, and floating-point addition is not associative, so the same inputs give slightly different outputs. And a result is only reproducible by someone else if they can get the same data, the same code and the same library versions. PyTorch's own notes start with the limit: completely reproducible results are not guaranteed across releases, commits or platforms, or between CPU and GPU, even with identical seeds (randomness.md 5-7).

Intuition

A pseudo-random generator is a long, fixed list of numbers, and the seed picks where in the list you start. Two runs with the same seed read the same numbers in the same order, so the same calls get the same values. Reproducibility breaks in three ways: a generator you forgot to seed starts somewhere unknown; an extra or missing draw shifts everything after it to a different place in the list; and a computation that does not use randomness at all still varies because the hardware schedules its additions differently. The fourth way is outside the program: the data file, the code or a library changed between the runs.

Mechanics

Seeding every generator

A PyTorch program usually draws from four kinds of generator. torch.manual_seed(seed) seeds the CPU generator and, through _manual_seed_impl, every CUDA, MPS, XPU and MTIA device and any custom backend, and returns the CPU Generator. Python's random and NumPy's global np.random are separate generators that PyTorch does not touch, so a script that wants all three repeatable seeds all three (randomness.md 25-72). torch.initial_seed() returns the seed in use, and get_rng_state / set_rng_state save and restore the generator's position, which is how a checkpoint resumes the same random stream.

A single global generator has a weakness: every draw moves everyone's position. Insert one extra random call (a new augmentation, a dropout layer, a debug sample) and every later draw comes from a different place in the stream, so the head's initial weights change even though you only touched the data pipeline. torch.random.fork_rng() runs a block and then restores the global state, so the block's draws do not shift what comes after. Passing an explicit torch.Generator to the code that needs randomness, one per purpose, goes further: each stream depends only on its own seed.

Data-loader workers

With num_workers > 0, each worker is a separate process. When an iterator starts, the loader draws one 64-bit _base_seed from loader.generator (or the global torch generator if none was given). Each worker then runs its seeding block: seed = base_seed + worker_id, then random.seed(seed), torch.manual_seed(seed), and, when NumPy is installed, np.random.seed(_generate_state(base_seed, worker_id)). _generate_state is a pure-Python copy of NumPy's SeedSequence hashing that turns (worker id, low and high 32 bits of base_seed) into four 32-bit words. Your worker_init_fn runs after this seeding and before data loading (dataloader.py 188-190), and get_worker_info() gives it the worker's id and seed.

So at this release every worker's global generators already differ, and they are repeatable if base_seed is. The order of the data is a separate draw: RandomSampler.__iter__ uses the loader's generator, and without one it draws a fresh seed from the global torch generator. PyTorch's recommended recipe passes a seeded torch.Generator as generator= and seeds random and NumPy from torch.initial_seed() in worker_init_fn (randomness.md 251-272). The trap that remains is a generator object you build yourself, for example self.rng = np.random.default_rng(0) in Dataset.__init__: each worker gets a copy of the same object in the same state, and the per-worker reseeding never touches it.

Nondeterministic kernels

Floating-point addition rounds after every step, so (a + b) + c and a + (b + c) can differ in the last bit. A GPU kernel that sums with atomic adds, such as scatter_add_ or the backward of an embedding or an index, adds contributions in whatever order threads arrive, which changes between runs. cuDNN adds another source: with torch.backends.cudnn.benchmark = True it times several convolution algorithms for each new input shape and keeps the fastest, and timing noise can pick a different one next run (randomness.md 73-92). Its default is False (cudnn/__init__.py 192-194).

torch.use_deterministic_algorithms(True) makes PyTorch pick a deterministic implementation where one exists and raise a RuntimeError for an operation that has none; its docstring lists both groups, for example index_add and scatter_add_ on CUDA switch, while AdaptiveAvgPool2d's CUDA backward and histc on CUDA raise. warn_only=True turns the errors into warnings so you can find every offender in one run. The same call turns on Inductor's deterministic mode, which skips on-device benchmarking for compiled code, and with it torch.utils.deterministic.fill_uninitialized_memory fills tensors from torch.empty with a known value (randomness.md 235-249). torch.backends.cudnn.deterministic = True (cudnn/__init__.py 221-229) is the older, cuDNN-only switch for convolutions. set_deterministic_debug_mode is the same setting under the names "default", "warn" and "error". Deterministic kernels are usually slower, and PyTorch says so in the docstring.

random_state in scikit-learn

Every scikit-learn estimator or splitter that uses randomness takes a random_state argument and passes it through check_random_state: None returns NumPy's global RandomState, an int builds a new RandomState(seed) each call, and a RandomState instance is returned as is; anything else raises ValueError. That difference matters. With an int, every call (every split, every fit) starts from the same place and gives the same result. With an instance, each call advances the shared generator, so calls differ from one another but the whole program is still repeatable. KFold shuffles with check_random_state(self.random_state), and at this release passing a random_state with shuffle=False raises an error because it would have no effect (_split.py 364-371).

Versioning data, code and environment

Seeds only reproduce a run on the same inputs. Record a content hash of the data (sha256 of each file, or of a sorted list of file hashes for a directory), the git commit of the code and whether the working tree was clean, and the exact environment: a lockfile with every package version, plus the Python, CUDA, cuDNN and driver versions and the GPU model, since kernels differ across them. Store these with the seed, the config and the metrics in one manifest next to each run's checkpoints. Tools such as DVC, lakeFS or a model registry do this at scale; the principle is that every input is named by its hash, not by a path that can change.

Worked example

Implementation

import hashlib, json, os, platform, random, subprocess
import numpy as np
import torch
from torch.utils.data import DataLoader, get_worker_info

def seed_everything(seed, deterministic=True):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)                      # CPU and every GPU
    torch.backends.cudnn.benchmark = False
    torch.use_deterministic_algorithms(deterministic, warn_only=False)

class Crops(torch.utils.data.Dataset):
    def __init__(self, images):
        self.images = images
        self.rng = None                          # built per worker, not here
    def __getitem__(self, i):
        if self.rng is None:
            info = get_worker_info()
            self.rng = np.random.default_rng(info.seed if info else torch.initial_seed())
        x, y = self.rng.integers(0, 32, size=2)
        return self.images[i][:, y:y + 224, x:x + 224]
    def __len__(self):
        return len(self.images)

def loader(dataset, seed):
    g = torch.Generator()
    g.manual_seed(seed)                          # data order and base_seed only
    return DataLoader(dataset, batch_size=64, shuffle=True, num_workers=4, generator=g)

def manifest(seed, data_paths, config):
    def sha256(path):
        with open(path, "rb") as f:
            return hashlib.file_digest(f, "sha256").hexdigest()
    git = lambda *a: subprocess.run(["git", *a], capture_output=True, text=True).stdout.strip()
    return {
        "seed": seed, "config": config,
        "data": {p: sha256(p) for p in sorted(data_paths)},
        "commit": git("rev-parse", "HEAD"), "dirty": bool(git("status", "--porcelain")),
        "torch": torch.__version__, "cuda": torch.version.cuda,
        "cudnn": torch.backends.cudnn.version(), "python": platform.python_version(),
        "gpu": torch.cuda.get_device_name() if torch.cuda.is_available() else None,
    }

# Some CUDA matmuls also need this set before the first CUDA call:
# os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

The snippet has not been run against PyTorch here; it only uses the calls linked above. The CUBLAS_WORKSPACE_CONFIG line comes from NVIDIA's cuBLAS documentation on reproducibility, not from PyTorch's docstring at this release, which does not mention it. hashlib.file_digest needs Python 3.11 or later.

Tradeoffs

Check yourself

• Why does self.rng = np.random.default_rng(0) in Dataset.__init__ give every worker the same augmentations, even though PyTorch reseeds NumPy in each worker?
• What two things does the DataLoader's generator argument control?
• Why can a GPU sum of the same tensor differ between runs with no randomness involved?
• When would you pass an int to random_state and when a RandomState instance?

Connections

Saving and restoring generator states is the RNG part of checkpointing. The worker processes and samplers are explained on data loading and batching, why float32 rounding depends on order on mixed precision, and how kernels are scheduled on GPU execution. Fixed splits matter for data splits and cross-validation, where comparing models across several seeds is the honest test. Dropout, one of the random draws, is on regularization.

References

• PyTorch v2.15.0: Reproducibility notes, manual_seed, fork_rng, _base_seed, _generate_state, worker seeding, RandomSampler.__iter__, use_deterministic_algorithms, set_deterministic_debug_mode, cuDNN flags.
• scikit-learn 1.9.1: check_random_state, KFold shuffling, the shuffle=False check.
• Goldberg. What Every Computer Scientist Should Know About Floating-Point Arithmetic. ACM Computing Surveys 23(1), 1991. doi:10.1145/103162.103163
• O'Neill. PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation. 2014; its seed-mixing design underlies NumPy's SeedSequence. pcg-random.org
• Bouthillier, Laurent and Vincent. Unreproducible Research is Reproducible. ICML 2019. PMLR 97