Generative adversarial networks

Problem

A generative model should produce new samples that look like the training data: images, audio, molecules. The variational autoencoder does this by maximising a bound on the log-likelihood, and the autoregressive models by factorising the likelihood exactly. Both need a density they can evaluate, and for images both pay for it: a VAE's Gaussian decoder averages over plausible outputs and blurs them, and an autoregressive model generates one pixel or token at a time.

The question a GAN asks instead is whether a network could be trained to produce samples directly, in one forward pass from random noise, judged only by whether they can be told apart from real ones. There is no formula for "looks real", so a second network learns one: the discriminator. The generator's loss is whatever the discriminator currently thinks of its samples, which makes training a two-player game rather than the minimisation of a fixed function, and most of what is hard about GANs follows from that.

Intuition

A forger and a detective. The detective is shown a mix of real banknotes and forgeries and learns what gives the forgeries away. The forger never sees a real note; it only learns, through the detective's verdict, which direction would have made a forgery more convincing. Each improves against the other, and if the game ends well the forgeries are indistinguishable and the detective can do no better than guess, outputting ½ for everything.

Two failure modes are already visible in the story. If the detective becomes too good too fast, its verdict on every forgery is "certainly fake" and says nothing about how to improve, so the forger stops learning. And the forger can fool a detective by perfecting one kind of note and never producing the others: this is mode collapse, and nothing in the detective's job of judging one sample at a time directly forbids it.

Mechanics

The minimax game

A generator G maps a noise vector z ~ p_z (usually a standard normal) to a sample G(z); the samples have some distribution p_g that is never written down. A discriminator D maps a sample to the probability that it is real. Goodfellow et al. defined the game

min_G max_D V(D, G) = 𝔼_x~pdata[log D(x)] + 𝔼_z~pz[log(1 − D(G(z)))]

The discriminator's half is ordinary binary cross-entropy, as on the loss functions page and in logistic regression: real samples labelled 1, generated samples labelled 0. Training alternates one or more gradient steps on D with one step on G, each using a minibatch of real data and a minibatch of noise; the generator's gradient flows back through D into G by ordinary backpropagation, with D's weights held fixed for that step.

The optimal discriminator and Jensen–Shannon

For a fixed generator, V is maximised pointwise. At each x the discriminator chooses y = D(x) to maximise p_data(x) log y + p_g(x) log(1 − y), and setting the derivative to zero gives

D*(x) = p_data(x) / (p_data(x) + p_g(x))
V(D*, G) = −log 4 + 2 · JS(p_data ‖ p_g)

where JS is the Jensen–Shannon divergence, the average KL divergence of each distribution from their mixture M = (p_data + p_g)/2. JS is zero only when the two distributions are equal, so against a perfect discriminator the generator's best move is p_g = p_data, where D* = ½ everywhere and V = −log 4 ≈ −1.3863. Unlike KL, JS is symmetric and bounded by log 2, and that bound is both a strength (it stays finite when supports differ) and, below, the source of a weakness.

The non-saturating generator loss

Early in training the generator's samples are poor and D rejects them confidently. Write D(x) = σ(a) with a the discriminator's logit. The generator's term log(1 − σ(a)) has derivative −σ(a) with respect to a, which goes to zero exactly when D is confident the sample is fake (a ≪ 0): the loss saturates where the generator most needs a signal. Goodfellow et al. therefore train G to maximise log D(G(z)) instead, whose derivative is 1 − σ(a), near 1 in the same region. This non-saturating loss has the same fixed point, and it is what almost every implementation uses: BCEWithLogitsLoss on D's logits for the fakes, with the target label set to 1.

Generators, discriminators and DCGAN

Any differentiable networks will do, but image GANs became reliable with the conventions of DCGAN (Radford et al.): a generator made of transposed convolutions that grow a 1×1 latent to an image, a convolutional discriminator with strided convolutions instead of pooling, batch normalisation in both (except the generator's output and the discriminator's input), ReLU in the generator with a Tanh output so pixels lie in [−1, 1], LeakyReLU with slope 0.2 in the discriminator, weights drawn from N(0, 0.02), and Adam with learning rate 2·10⁻⁴ and β₁ = 0.5. A transposed convolution's output size is

H_out = (H_in − 1) · stride − 2 · padding + dilation · (kernel − 1) + output_padding + 1

so kernel 4, stride 2, padding 1 exactly doubles the size, and kernel 4, stride 1, padding 0 turns 1×1 into 4×4: 1 → 4 → 8 → 16 → 32 → 64 in DCGAN's five layers.

Wasserstein critics and Lipschitz constraints

JS has a blind spot. If p_data and p_g do not overlap, which is typical early on when both live on thin low-dimensional sets inside pixel space, then JS = log 2 no matter how far apart they are, and a perfect discriminator's gradient carries no information about which direction is closer. Arjovsky et al. replaced it with the Wasserstein-1 (earth mover's) distance, the least total mass × distance needed to move one distribution onto the other, which shrinks smoothly as the generator approaches the data. By Kantorovich–Rubinstein duality

W₁(p_data, p_g) = sup_{‖f‖L ≤ 1} 𝔼_pdata[f(x)] − 𝔼_pg[f(x)]

so a critic f with no sigmoid, constrained to be 1-Lipschitz, is trained to maximise the gap in mean scores and the generator to minimise it. The constraint is the hard part. WGAN clipped every weight to a small box, which works poorly; WGAN-GP (Gulrajani et al.) instead adds λ(‖∇_x̂f(x̂)‖ − 1)² at random interpolations x̂ between real and fake samples, with λ = 10 and several critic steps per generator step; computing that penalty needs the gradient of an output with respect to an input, kept in the graph so it can itself be differentiated: torch.autograd.grad(..., create_graph=True). Spectral normalisation (Miyato et al.) bounds the Lipschitz constant layer by layer instead, dividing each weight matrix by its largest singular value, estimated with one power iteration per forward pass.

Convergence and regularisation

Gradient descent on a game is not gradient descent on a loss. Mescheder et al. made this concrete with the Dirac-GAN: real data is a single point at 0, the generator is a single point θ, and the discriminator is D(x) = σ(ψx). The only equilibrium is θ = ψ = 0, but near it the two players' updates rotate around it rather than descending into it, and simultaneous gradient steps of any fixed size spiral outward. The paper shows that unregularised GANs, and also WGAN and WGAN-GP, are not guaranteed to converge on this example, and that adding the R1 penalty γ/2 · 𝔼_pdata‖∇_xD(x)‖², a zero-centred gradient penalty on real data only, makes it converge. R1 and spectral normalisation are the stabilisers most later image GANs use, along with separate learning rates for D and G (the two time-scale update rule of Heusel et al.).

PyTorch has no GAN class or GAN loss: a GAN is two ordinary modules, two optimizers and the loop that alternates them, which is why the table is a list of building blocks.

Worked example

Implementation

A DCGAN for 64×64 RGB images, trained with the non-saturating loss and an R1 penalty on real images. The discriminator ends in a logit, not a probability, so the loss is BCEWithLogitsLoss.

import torch
from torch import nn

nz, ngf, ndf = 100, 64, 64

def up(c_in, c_out, stride=2, padding=1):
    return [nn.ConvTranspose2d(c_in, c_out, 4, stride, padding, bias=False),
            nn.BatchNorm2d(c_out), nn.ReLU(True)]

def down(c_in, c_out, norm=True):
    layers = [nn.Conv2d(c_in, c_out, 4, 2, 1, bias=False)]
    if norm:
        layers.append(nn.BatchNorm2d(c_out))
    return layers + [nn.LeakyReLU(0.2, True)]

G = nn.Sequential(                       # (N, nz, 1, 1) -> (N, 3, 64, 64)
    *up(nz, ngf * 8, stride=1, padding=0),           # 4x4
    *up(ngf * 8, ngf * 4), *up(ngf * 4, ngf * 2),    # 8x8, 16x16
    *up(ngf * 2, ngf),                               # 32x32
    nn.ConvTranspose2d(ngf, 3, 4, 2, 1, bias=False), # 64x64
    nn.Tanh(),
)
D = nn.Sequential(                       # (N, 3, 64, 64) -> (N, 1) logits
    *down(3, ndf, norm=False), *down(ndf, ndf * 2),
    *down(ndf * 2, ndf * 4), *down(ndf * 4, ndf * 8),
    nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False), nn.Flatten(),
)

def init_weights(m):
    if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
        nn.init.normal_(m.weight, 0.0, 0.02)

G.apply(init_weights)
D.apply(init_weights)
bce = nn.BCEWithLogitsLoss()
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))

def train_step(real, gamma=10.0):
    z = torch.randn(real.size(0), nz, 1, 1)
    fake = G(z)

    real = real.requires_grad_(True)
    logit_real, logit_fake = D(real), D(fake.detach())
    loss_d = (bce(logit_real, torch.ones_like(logit_real))
              + bce(logit_fake, torch.zeros_like(logit_fake)))
    grad, = torch.autograd.grad(logit_real.sum(), real, create_graph=True)
    r1 = grad.pow(2).flatten(1).sum(1).mean()
    opt_d.zero_grad()
    (loss_d + gamma / 2 * r1).backward()
    opt_d.step()

    logit = D(fake)                          # non-saturating: label fakes real
    loss_g = bce(logit, torch.ones_like(logit))
    opt_g.zero_grad()
    loss_g.backward()
    opt_g.step()
    return loss_d.item(), loss_g.item()

fake.detach() keeps the discriminator's loss from reaching G. The generator step still backpropagates through D, which also leaves gradients in D's parameters; they are harmless because opt_d.zero_grad() clears them before D's next update. For a Wasserstein critic, drop the BCE and read D's output as an unbounded score: the critic loss is f(fake).mean() - f(real).mean() plus the gradient penalty at interpolates, the generator loss is -f(fake).mean(), and batch norm in the critic is replaced (Gulrajani et al. suggest layer norm) because the penalty is defined per sample. Spectral norm is one call per layer: torch.nn.utils.parametrizations.spectral_norm(nn.Conv2d(...)).

Tradeoffs

• Sharp, fast samples. One forward pass per sample and no pixel-wise likelihood that rewards averaging, so GAN images are sharp and generation is far cheaper than iterative samplers. The same objective gives no likelihood to evaluate, compare or use for anomaly scores.
• Mode collapse. The generator can cover part of the data well and ignore the rest; the reverse-KL-like behaviour in the worked example is the mild form. Minibatch statistics in the discriminator, gradient penalties and larger discriminators reduce it, but none rules it out.
• Unstable training. A game has no loss that decreases; D and G losses oscillate, and neither one tracks sample quality. Regularisers (R1, spectral norm), balanced learning rates and checkpoint selection by FID are standard, and training still needs more tuning than a likelihood model.
• Evaluation. FID is the default but depends on the Inception network, the sample count and preprocessing, and it compresses quality and diversity into one number. Precision and recall for generative models separate the two.
• Conditioning and editing. Class labels or text can be fed to both networks (conditional GANs), and a well-trained generator's latent space supports smooth interpolation and editing, which likelihood models do not offer as directly.
• Where they are used now. For general text-to-image generation, diffusion models have largely replaced GANs on diversity and training stability. Adversarial losses survive as components: the discriminator loss in the decoders of image and audio tokenizers, super-resolution, and distilling slow samplers into few-step ones.

Connections

The discriminator is a logistic regression on learned features, trained with the binary cross-entropy from loss functions, and the generator learns entirely through backpropagation into that classifier. The divergences in the worked example are the ones on entropy and KL. The variational autoencoder is the likelihood-based alternative with the opposite failure mode (blur rather than dropped modes), and the autoencoder page covers the decoder that a generator resembles. The architecture is built from convolutional networks and activation functions, initialised as on initialization, and trained with the Adam settings explained on optimizers.

References

• Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio. Generative Adversarial Nets. NeurIPS, 2014 (arXiv 1406.2661; the game, D*, the JS result and the non-saturating loss).
• Radford, Metz and Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. ICLR, 2016 (arXiv 1511.06434; DCGAN).
• Arjovsky and Bottou. Towards Principled Methods for Training Generative Adversarial Networks. ICLR, 2017 (arXiv 1701.04862; disjoint supports and saturation).
• Arjovsky, Chintala and Bottou. Wasserstein Generative Adversarial Networks. ICML, 2017 (arXiv 1701.07875).
• Gulrajani, Ahmed, Arjovsky, Dumoulin and Courville. Improved Training of Wasserstein GANs. NeurIPS, 2017 (arXiv 1704.00028; WGAN-GP).
• Heusel, Ramsauer, Unterthiner, Nessler and Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS, 2017 (arXiv 1706.08500; TTUR and FID).
• Miyato, Kataoka, Koyama and Yoshida. Spectral Normalization for Generative Adversarial Networks. ICLR, 2018 (arXiv 1802.05957).
• Mescheder, Geiger and Nowozin. Which Training Methods for GANs do actually Converge? ICML, 2018 (arXiv 1801.04406; the Dirac-GAN and R1).
• Chong and Forsyth. Effectively Unbiased FID and Inception Score and Where to Find Them. CVPR, 2020 (arXiv 1911.07023; FID's sample-size bias).
• PyTorch v2.15.0: torch/nn/modules/loss.py, torch/autograd/__init__.py, torch/nn/utils/parametrizations.py.