Overfitting and underfitting

Problem

The generalization page defined the gap between training error and error on new data. When a model does badly, that gap is the first thing to look at, because it points to opposite fixes. Add capacity to a model that is too simple and it improves. Add capacity to a model that already memorizes and it gets worse.

So the working problem is diagnosis: from numbers you can measure on a training set and a validation set, decide which situation you are in before you spend time on the wrong remedy.

Intuition

A student who reads only the chapter titles fails every test (underfit: not enough learned). A student who memorizes the answers to last year's exam scores perfectly on it and badly on this year's (overfit: learned the specific questions, not the subject). The first student's practice scores and real scores are both low. The second student's practice score is high and real score is low.

That is the whole diagnostic. With a noise floor σ² that no model can beat:

• Underfitting (high bias). Training error is high, close to validation error, and both sit well above the noise floor. More data does not help; more capacity does.
• Overfitting (high variance). Training error is low, validation error is much higher, and the gap shrinks as the training set grows. More data, a simpler model or regularization help.
• A good fit. Both errors are close to the noise floor and close to each other.

Training error alone cannot tell you any of this, since a flexible enough model drives it to zero. The signal is always the comparison with held-out data (see data splits and cross-validation).

Errors above 0.5 are clipped at the top. The line's curves meet high; the degree-9 curves meet low, but only after enough data.

Mechanics

Two curves that diagnose

A learning curve plots training and validation error against the number of training rows. It answers "would more data help?": if the two curves have already met, the answer is no; if they are still apart, yes. A validation curve (also called a complexity curve) plots the same two errors against one hyperparameter, such as polynomial degree, tree depth or regularization strength, with the data fixed. Underfitting is the left side, where both are high; overfitting is the right side, where training keeps falling and validation turns up. Both are computed with cross-validation, so the validation number comes from rows the model was not fitted on.

Left of the minimum the model overfits; right of it, it underfits. Values above 0.6 are clipped.

Remedies by diagnosis

• For underfitting: a more flexible model (more features, deeper trees, more layers), less regularization, or better features; train longer if the optimizer has not converged.
• For overfitting: more (or augmented) data, a simpler model, regularization (an L2 or L1 penalty, dropout), early stopping, or ensembling to average away variance.
• For both: check the labels and the split first. Noisy labels raise the floor, and leakage can make a model look better than it is.

Memorization

Overfitting at its extreme is memorization: a model with enough capacity can fit labels that are pure noise. Deep networks have been shown to reach zero training error on randomly shuffled labels (Zhang et al., 2017), which is why low training error says nothing about whether anything was learned. The worked example does the same with a decision tree. Regularization and early stopping work by making this memorization costly or by stopping before it happens.

Early stopping

For models trained iteratively, such as boosted trees or neural networks, the number of training steps acts like a capacity knob: early on the model is simple, and every step lets it fit more detail. Watching validation error and stopping at its minimum picks the right capacity automatically. scikit-learn's validation_curve and learning_curve do the bookkeeping for the other cases.

Worked example

Implementation

import numpy as np
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge

model = make_pipeline(PolynomialFeatures(9), StandardScaler(), Ridge(alpha=1e-2))

# does more data help?
sizes, train, val = learning_curve(
    model, X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 6),
    scoring="neg_mean_squared_error",
)
print(-train.mean(axis=1), -val.mean(axis=1))   # rows: sizes, columns: folds

# which capacity is right?
alphas = np.logspace(-4, 1, 6)
train, val = validation_curve(
    model, X, y, param_name="ridge__alpha", param_range=alphas,
    cv=5, scoring="neg_mean_squared_error",
)

The scores are returned negated for losses (neg_mean_squared_error), so higher is better; flip the sign before reading them as errors. Both functions refit the model many times, so on large data use a subsample or fewer sizes.

Tradeoffs

• Diagnosis needs honest validation data. If the validation set is tuned against repeatedly, it leaks into the model and the gap looks smaller than it is. Keep a final test set untouched.
• Noise changes the picture. With noisy labels the floor is above zero, so "training error is not zero" is not a sign of underfitting; compare with the floor, not with 0.
• Remedies cost something. More data is slow or expensive; regularization adds a hyperparameter; early stopping needs a validation split that the model then cannot train on; simpler models may underfit.
• Modern models blur the line. Very large networks can interpolate their training data and still generalize, so a zero training error does not always mean the usual overfitting (see double descent on the generalization page). Judge by held-out performance.
• Curves are noisy. One split gives a jagged curve. Use cross-validation and look at the trend, as the medians above do.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, chapter 7: model assessment and selection.
• Zhang, Bengio, Hardt, Recht and Vinyals. Understanding deep learning requires rethinking generalization. ICLR 2017. arXiv:1611.03530.
• Belkin, Hsu, Ma and Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS 2019. arXiv:1812.11118.
• scikit-learn 1.9.1: learning_curve, validation_curve.