Autoencoders

Problem

Most data has far fewer degrees of freedom than coordinates. A 28×28 digit has 784 pixels, but the set of images that look like digits is a thin sliver of that space: stroke thickness, slant, style. We often have plenty of unlabeled examples and want a short description of each one, to compress it, to remove noise, to spot an input that does not fit, or to give a later model better features than raw pixels.

There is no label to supervise that description. An autoencoder uses the input itself as the target, which makes this a self-supervised problem in the sense of learning paradigms: learn an encoder that maps x to a code z, a decoder that maps z back to x̂, and train both so that x̂ is close to x. The interesting part is everything that stops the network from cheating.

Intuition

Imagine describing a photo to a friend over a phone line that only carries sixteen numbers, and they have to redraw it. You cannot send pixels, so you agree on a vocabulary: where the object is, how big, what tilt. The encoder and decoder learn that vocabulary together, and because the line is narrow, the vocabulary must capture the directions in which the data actually varies. Noise and rare details are the first things dropped.

If the line were as wide as the image, the pair could agree to send the pixels unchanged and learn nothing. Every useful autoencoder therefore has some constraint: a narrow bottleneck (undercomplete), corrupted inputs it must clean up (denoising), or a penalty that keeps most code units silent (sparse). The constraint is what turns copying into learning.

The bottleneck forces a short code; the loss needs no label because the input is its own target.

Mechanics

The objective

An encoder f_θ and a decoder g_φ are trained to minimize the average reconstruction loss over the data:

z = f_θ(x),   x̂ = g_φ(z),   L(θ, φ) = (1/N) Σ_n ℓ(x_n, g_φ(f_θ(x_n)))

For real-valued data ℓ is squared error, PyTorch's MSELoss, which averages over every element by default. For pixels scaled to [0, 1] binary cross-entropy is common, with a sigmoid at the decoder's output or, more stably, BCEWithLogitsLoss on the raw outputs. Both are the negative log-likelihood of the input under a Gaussian or Bernoulli decoder, the same correspondence as in loss functions. Training is an ordinary training loop in which the target is the batch itself.

A linear autoencoder is PCA

Take centred data, a linear encoder z = Ex with E of shape (k, d), a linear decoder x̂ = Dz with D of shape (d, k), k < d, and squared error. The reconstruction is x̂ = DEx, a matrix of rank at most k, and the best rank-k map in squared error is the orthogonal projection onto the top k eigenvectors of the data's second-moment matrix Σ = (1/N) Σ_n x_nx_nᵀ. That is exactly PCA. Baldi and Hornik showed that this loss has no local minima other than the global one: every other critical point is a saddle, so gradient descent reaches the PCA subspace.

The minimum loss is the variance the code throws away. With error averaged over d coordinates, it is (λ_k+1 + … + λ_d)/d. But the learned E and D are not the eigenvectors themselves: any invertible A gives the same product (DA⁻¹)(AE), so the encoder rows can be any basis of the subspace, scaled and mixed. The product DE is unique, the factors are not. Non-linear encoders and decoders can follow curved data that PCA cannot, which is the reason to use a neural autoencoder at all (Hinton and Salakhutdinov's 2006 deep autoencoders beat PCA on images and documents this way).

The one-unit code can only say "how far along u"; everything perpendicular to u is lost, and that loss is exactly PCA's.

Why a wide autoencoder learns nothing, and three fixes

If the code is as wide as the input, E = D = I is a perfect solution, and a large enough non-linear network can approximate it whatever the code size. Three standard constraints rule it out:

• Undercomplete. Make the code smaller than the input, as above. Simple, but the right size is a guess and too small loses detail.
• Denoising. Corrupt the input (Gaussian noise, or masking a fraction of entries) and train x̂ = g(f(x̃)) to match the clean x. The identity now scores badly, because it reproduces the noise. For a linear map and Gaussian noise of variance σ², the best map is Σ(Σ + σ²I)⁻¹: in each eigendirection it multiplies by λ/(λ + σ²), keeping directions with a lot of signal and shrinking the weak ones. Vincent et al. (2008) introduced the idea; masked language and image models are its descendants.
• Sparse. Allow a wide code but add a penalty, such as λ·Σ|z_j| (an L1 penalty from regularization) or a KL divergence pulling each unit's average activation toward a small target, so each input uses only a few units. Sparse autoencoders on a model's activations are a current tool for finding interpretable features inside large language models.

Convolutional autoencoders and output_padding

For images the encoder is a stack of strided convolutions and the decoder mirrors it with transposed convolutions, which map each input position to a k×k patch of the output, summing where patches overlap. A Conv2d has output size ⌊(H + 2p − d(k − 1) − 1)/s⌋ + 1, and a ConvTranspose2d with the same arguments has (H − 1)s − 2p + d(k − 1) + output_padding + 1, so they invert each other's shapes (not their values). With stride 2 the floor makes Conv2d map both 13 and 14 to 7, so going back is ambiguous: k = 3, s = 2, p = 1 gives 13 from 7, and output_padding=1 picks 14. PyTorch's docs stress that output_padding only selects the output shape and adds no zeros. Alternatively, pass output_size to forward. Transposed convolutions with stride that does not divide the kernel can leave checkerboard artifacts; upsampling then an ordinary convolution is a common alternative.

Worked example

Gradient descent finds PCA's subspace but not PCA's basis; denoising finds a soft version of the same subspace without any bottleneck.

Implementation

import torch
import torch.nn as nn

class ConvAutoencoder(nn.Module):
    def __init__(self, code=16):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, 3, stride=2, padding=1), nn.ReLU(),    # 16x14x14
            nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(),   # 32x7x7
            nn.Flatten(),
            nn.Linear(32 * 7 * 7, code),
        )
        self.decoder = nn.Sequential(
            nn.Linear(code, 32 * 7 * 7), nn.ReLU(),
            nn.Unflatten(1, (32, 7, 7)),
            nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1, output_padding=1), nn.ReLU(),
            nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1),  # logits
        )

    def forward(self, x):
        return self.decoder(self.encoder(x))

model = ConvAutoencoder()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.BCEWithLogitsLoss()        # pixels in [0, 1]

for x, _ in loader:                     # labels are ignored
    noisy = (x + 0.3 * torch.randn_like(x)).clamp(0, 1)   # drop for a plain autoencoder
    loss = loss_fn(model(noisy), x)     # the target is the clean input
    opt.zero_grad()
    loss.backward()
    opt.step()

with torch.no_grad():
    codes = model.encoder(x)            # (N, 16) features for search, clustering, anomalies
    images = torch.sigmoid(model(x))

The worked example's table counts a sigmoid output, so its parameter total equals this model's. For anomaly detection, train on normal data only and flag inputs whose reconstruction error is far above the training distribution; but check, because autoencoders sometimes reconstruct unfamiliar inputs well. To reuse the encoder as features for a classifier, freeze it or fine-tune it with a small head and keep the data splits clean.

Tradeoffs

• Autoencoder versus PCA. If the data is close to a linear subspace, PCA gives the same answer in closed form with ordered, orthogonal components. A neural autoencoder pays off only when the structure is curved, and costs training, tuning and an arbitrary basis.
• Code size. Too small and reconstructions blur; too large and, without another constraint, the network copies. Denoising and sparsity let you choose a generous size safely.
• Blurry outputs. Squared error is minimized by the average of plausible outputs, so fine detail is smoothed away. Perceptual or adversarial losses sharpen results at the cost of stability.
• Not a generator. A plain autoencoder's code space has holes: decoding a random z, or the midpoint of two codes, often gives garbage because nothing shaped the space between training codes. Variational autoencoders add exactly that shaping.
• Features without labels. Reconstruction is a cheap, label-free objective, but low reconstruction error does not guarantee codes that are useful for a downstream task; contrastive or masked objectives often give better features.

Connections

References

• Bourlard and Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 1988.
• Baldi and Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 1989.
• Hinton and Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 2006.
• Vincent, Larochelle, Bengio and Manzagol. Extracting and Composing Robust Features with Denoising Autoencoders. ICML 2008.
• Ng. Sparse autoencoder. CS294A lecture notes, Stanford, 2011.
• Dumoulin and Visin. A guide to convolution arithmetic for deep learning. 2016. arXiv:1603.07285
• Odena, Dumoulin and Olah. Deconvolution and Checkerboard Artifacts. Distill, 2016. distill.pub/2016/deconv-checkerboard
• PyTorch v2.15.0: torch/nn/modules/conv.py, torch/nn/modules/linear.py, torch/nn/modules/loss.py.