The bias–variance tradeoff

Problem

The overfitting page said that too simple a model underfits and too flexible a model overfits. That is a description. To reason about it, to say why an ensemble helps or why a penalty helps, we need the error written as parts that each respond differently to the choices we make.

The question: for a fixed query point, if we could redraw the training set many times and refit, what does the average squared error consist of?

Intuition

Picture a dart player throwing many rounds at one target. Bias is how far the centre of the cluster of darts is from the bullseye. Variance is how spread out the darts are around their own centre. A steady player who aims a little left has high bias and low variance; a shaky player who aims correctly on average has low bias and high variance. Noise is a target that itself jitters: even a perfect player misses by that much.

Each training set is one round of throws, and the fitted model's prediction at a point is the dart. A straight line fitted to a sine wave lands in nearly the same wrong place every time (high bias, low variance). A degree-9 polynomial tracks whichever noise it was shown (low bias, high variance). Ways to cut variance without raising bias much, such as averaging many models or penalizing large coefficients, are what make flexible models usable.

Bias² falls as the degree rises while variance climbs; the ridge penalty pulls the degree-9 variance back down.

Mechanics

The decomposition

Let the data be y = f(x) + ε with E[ε] = 0 and Var(ε) = σ². Let f̂ be the model fitted on a random training set D. At a fixed x, the expectation below is over D and over the noise in the test label y:

E[(y − f̂(x))²]
  = E[(f(x) + ε − f̂)²]
  = σ²  +  E[(f(x) − f̂)²]                      (ε is independent of f̂ and has mean 0)
  = σ²  +  (f(x) − E[f̂])²  +  E[(f̂ − E[f̂])²]  (add and subtract E[f̂]; the cross term is 0)
  = noise + bias² + variance

Two steps do all the work. First, the noise in the new label is independent of anything the model learned, so its contribution separates. Second, adding and subtracting the average prediction E[f̂] makes the cross term vanish because E[f̂ − E[f̂]] = 0. Averaging over x as well gives a single number for each part. The result holds for squared error; other losses do not split this cleanly (see loss functions). The expectation and variance page has the rules used here.

Bias is the gap between a model family's average fit and the truth, not the error of any one fit.

Measuring it

On real data f is unknown and there is only one training set, so the three parts cannot be read off directly. They can be estimated when the data are simulated, as on this page: draw many training sets, fit each, and at each test x compute the mean prediction, the squared gap of that mean from f (bias²), and the spread of the predictions around it (variance). On real data the practical stand-ins are curves, covered next.

Reading the parts from curves

• High bias. Training and validation error are both high and close together. A learning curve is flat: more rows do not help, because the model family cannot represent the pattern.
• High variance. Training error is low, validation error is far above it. The learning curve's gap narrows as rows are added, since more data pins the fit down.
• The validation curve over complexity is U shaped: the left arm is bias, the right arm is variance, the floor is noise plus whatever the best model leaves.

The fixes are asymmetric. Variance falls with more data, simpler models, regularization and averaging many models. Bias falls with richer models and better features, and does not fall with more data.

Averaging cuts variance

If M predictions each have variance v and pairwise correlation ρ, their average has variance ρv + (1 − ρ)v/M. Averaging leaves the bias of each member unchanged (the mean of the average is the average of the means) and shrinks the second term towards zero as M grows. Bagging makes the members differ by fitting each on a bootstrap resample of the rows; the more decorrelated the members, the more variance disappears. Boosting works the other way, mostly reducing bias, as the gradient boosting page showed.

Worked example

Implementation

import numpy as np
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

rng = np.random.default_rng(0)
grid = np.linspace(0.0125, 0.9875, 40).reshape(-1, 1)
truth = np.sin(2 * np.pi * grid[:, 0])

preds = []
for _ in range(200):                       # 200 fresh training sets
    x = rng.uniform(0, 1, (25, 1))
    y = np.sin(2 * np.pi * x[:, 0]) + rng.normal(0, 0.3, 25)
    model = make_pipeline(PolynomialFeatures(3), Ridge(alpha=1e-3)).fit(x, y)
    preds.append(model.predict(grid))
preds = np.array(preds)                    # (200, 40)

bias2 = ((preds.mean(axis=0) - truth) ** 2).mean()
variance = preds.var(axis=0).mean()
print(bias2, variance, 0.3 ** 2)           # error = sum of the three

This only works when the truth is known. For real data, fit with learning_curve and validation_curve from sklearn.model_selection and read the diagnosis from the shapes described above.

Tradeoffs

• Squared error only. The clean three-way split needs squared loss. Classification with 0–1 loss has several competing decompositions, and none is as tidy.
• Not a strict ladder. The "U" shape is typical for classical models but not universal. Very large networks and boosted ensembles can reach zero training error and still improve on new data as they grow (double descent; see generalization), so variance does not always rise with capacity.
• Hard to measure for real. With one dataset you cannot redraw training sets; bootstrap resampling approximates it, and learning curves give indirect evidence. The numbers on this page come from a known truth.
• Remedies trade against each other. A bigger penalty cuts variance and raises bias; the best setting is found by validation, not by theory.
• Even-degree polynomials look odd here. The sine is an odd function around the middle of the interval, so degree 2 adds variance and no bias reduction over degree 1; the table sticks to odd degrees.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, section 7.3: the bias–variance decomposition.
• Geman, Bienenstock and Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 1992.
• Belkin, Hsu, Ma and Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS 2019. arXiv:1812.11118.
• Breiman. Bagging predictors. Machine Learning, 1996.
• scikit-learn 1.9.1: learning_curve, validation_curve.