Regularization

Problem

The overfitting page showed that a flexible model fits noise, and the bias–variance page showed why: its predictions swing from one training set to the next. Shrinking the model by hand (fewer features, shallower trees) is a blunt tool, and for neural networks there is no obvious knob.

Regularization is a family of smoother knobs. Each one gives up a controlled amount of training fit (adds a little bias) to cut variance, and the strength is a hyperparameter chosen by validation.

Intuition

Think of the training loss as a bowl and the model parameters as a ball resting at the bottom. With noisy data the bottom sits at a place that exploits the noise: some weights are large, balanced against each other. A penalty adds a second bowl centred at zero and the ball settles between them. The stronger the penalty, the closer to zero.

• L2 (ridge, weight decay) penalizes squared weights. It shrinks all weights smoothly, rarely to exactly zero, and handles correlated features by splitting weight among them.
• L1 (lasso) penalizes absolute values. It can set weights exactly to zero, so it also selects features.
• Dropout randomly removes units during training, so no unit can rely on a particular partner being present.
• Early stopping, data augmentation and label smoothing regularize without touching the weights directly: they limit how far training goes, enlarge the dataset with label-preserving changes, or soften the targets.

The penalized fit is where the loss contours first touch the constraint region. A circle has no corners; a diamond's corners lie on the axes. The picture is schematic.

Mechanics

L2: ridge and weight decay

Add λ‖w‖² (or ½λ‖w‖², depending on convention) to the loss. For linear regression the minimizer has a closed form that differs from ordinary least squares by one term on the diagonal:

w = (XᵀX + αI)⁻¹ Xᵀy

The αI makes the matrix invertible even with correlated or duplicated features, and shrinks each weight along the directions the data barely constrains (those with small singular values) the most. scikit-learn's _solve_cholesky literally adds α to the diagonal of XᵀX and solves. For gradient-trained models the penalty's gradient is λw, so each step pulls every weight towards zero in proportion to its size. That is why L2 is also called weight decay. The linear regression page has the full derivation and path.

L1: lasso and sparsity

Add λ‖w‖₁ instead. The penalty has a corner at zero, so the minimizer sets a weight exactly to zero whenever the data's pull on it is weaker than λ. Coordinate descent updates one weight at a time using the soft-threshold function S(ρ, λ) = sign(ρ)·max(|ρ| − λ, 0), which is exactly where the zeros come from. Elastic net mixes the two with a ratio l1_ratio; scikit-learn's objective is 1/(2n)‖y − Xw‖² + α·l1_ratio·‖w‖₁ + ½α(1 − l1_ratio)‖w‖². Note that scikit-learn divides the squared error by 2n, so its α is on a different scale from Ridge's.

Each line is one coefficient. Grey lines reach zero and stay there; the informative ones shrink gradually and drop out last.

Dropout

During training, each unit's output is set to zero with probability p, and the survivors are multiplied by 1/(1 − p) ("inverted" dropout). The scaling keeps the expected output unchanged, so at evaluation time the layer is simply the identity and nothing needs rescaling. Because a different random subnetwork is trained at every step, a unit cannot depend on another being present; the effect resembles averaging an enormous ensemble of thinned networks, though with shared weights. PyTorch's nn.Dropout checks 0 ≤ p ≤ 1 in its constructor and calls F.dropout(input, p, self.training, ...), so model.eval() switches it off. Forgetting eval() at inference time is a common bug that makes predictions noisy.

Weight decay versus an L2 penalty under Adam

With plain SGD, adding λw to the gradient and shrinking the weight by a factor (1 − ηλ) are the same thing. With an adaptive optimizer like Adam they are not. Adam divides the update by the running root-mean-square of the gradient, and if the penalty is folded into the gradient, the decay is divided too: weights whose loss gradients are large are decayed little, and weights with tiny gradients are decayed a lot. AdamW (Loshchilov and Hutter, 2019) applies the decay directly to the weights, w ← w(1 − lr·λ), outside the adaptive step, so every weight decays at the same rate. In PyTorch v2.15.0, AdamW is Adam with decoupled_weight_decay=True; the branch is in adam.py. See the optimizers page for the rest of Adam.

Early stopping, augmentation, label smoothing

Early stopping halts training at the minimum of validation error (see the overfitting page); for linear models it behaves much like an L2 penalty. Data augmentation adds label-preserving transformed copies of the data (crops, flips, noise) so the model sees more variety: it regularizes by enlarging the training set, not by shrinking weights. Label smoothing replaces a one-hot target with (1 − ε) on the true class and ε/(K − 1) elsewhere, which stops the network from pushing its logits to infinity to reach a probability of exactly 1.

Worked example

Implementation

import torch
from torch import nn
from sklearn.linear_model import Lasso, Ridge

# sklearn: closed-form ridge, coordinate-descent lasso
ridge = Ridge(alpha=20.0).fit(X, y)
lasso = Lasso(alpha=0.4).fit(X, y)
print((lasso.coef_ != 0).sum(), "features kept")

# PyTorch: dropout in the model, decoupled decay in the optimizer
model = nn.Sequential(
    nn.Linear(64, 128), nn.ReLU(), nn.Dropout(p=0.1),
    nn.Linear(128, 10),
)
# no decay on biases and norm scales is the common convention
decay = [p for n, p in model.named_parameters() if p.ndim > 1]
no_decay = [p for n, p in model.named_parameters() if p.ndim <= 1]
opt = torch.optim.AdamW(
    [{"params": decay, "weight_decay": 0.01},
     {"params": no_decay, "weight_decay": 0.0}],
    lr=1e-3,
)

model.train()   # dropout active
# ... training steps ...
model.eval()    # dropout off, identity

Always standardize features before an L1 or L2 penalty: the penalty treats all weights alike, so features on a larger scale would otherwise be penalized less for the same effect.

Tradeoffs

• Strength is a hyperparameter. Too weak and nothing changes; too strong and the model underfits (the right arm of the validation curve). Choose it by cross-validation, on a log grid.
• Lasso is unstable with correlated features. Among a group of near-duplicates it keeps one, and which one can change with the sample. Elastic net or ridge keeps the group.
• Shrinkage biases the kept weights. Lasso's survivors are smaller than their true values, as the worked example shows; a common fix is to refit an unpenalized model on the selected features.
• Dropout is noisy. It slows convergence, and it interacts with normalization layers (batch statistics change between training and evaluation). It matters less in large, data-rich transformer pretraining, where it is often set to 0.
• Penalties do not fix bad data. If the validation set leaks, regularization strength chosen on it will be too weak; see data leakage.
• Which weights to decay. Biases and normalization gains are usually left out of weight decay.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, sections 3.4 (shrinkage) and 3.8 (lasso and elastic net).
• Tibshirani. Regression shrinkage and selection via the lasso. JRSS B, 1996.
• Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR 2014.
• Loshchilov and Hutter. Decoupled weight decay regularization. ICLR 2019. arXiv:1711.05101.
• scikit-learn 1.9.1: _solve_cholesky, ElasticNet.fit.
• PyTorch v2.15.0: Dropout, decoupled weight decay in adam.py.