Train, validation and test splits

Problem

Generalization is the goal, and held-out data is how you measure it. The catch is that every decision you make by looking at held-out scores, such as which model, which settings, when to stop, quietly trains on those scores. Do enough of that and the held-out number is as optimistic as the training one. You need data that has had no influence on any decision.

The second catch is that "held out" only means something if the held-out rows are really independent of the training rows. Several records from the same patient, or tomorrow's data shuffled in with yesterday's, make the test too easy, and the score you report is not the one you will get in use.

Intuition

Think of three kinds of practice. Training data is the homework you study from. Validation data is a set of practice exams you take repeatedly to decide how to study: which book, how many hours. Test data is the real exam, sealed in an envelope until the end. Once you have peeked at a practice exam to adjust your study plan, it no longer tells you how you will do on something fresh, which is why the envelope must stay shut.

And the exam must resemble reality. If the real exam will cover next year's material, practising on a shuffled mix of this year's and next year's is cheating, even if the shuffle was random.

The 70/15/15 proportions are a common default, not a rule; the section on sizes below says what really matters.

Mechanics

Why three splits

• Train fits the model's parameters.
• Validation (also called dev) compares models and settings: architecture, regularization strength, learning rate, early stopping point. You look at it many times, so it becomes slightly fitted to.
• Test is used once, after all choices are final, to report an estimate of performance on new data.

With only train and test, any tuning you do uses the test set, and the final score is biased upward. When data is scarce, cross-validation replaces the single validation set by rotating many of them, but the test set stays untouched either way.

Split sizes

The right sizes depend on how precisely you need to know the score, not on a ratio. An accuracy measured on m independent test examples has standard error √(p(1 − p) / m), so for p = 0.8 it is 0.040 at m = 100, 0.013 at m = 1,000 and 0.004 at m = 10,000. Quadrupling the test set only halves the uncertainty. With a million examples, 1% held out is already ample; with a hundred, a split of any ratio leaves a noisy estimate and cross-validation is the better tool. The training part should be as large as you can spare, since more data lowers the generalization gap.

Stratified splits

A random split keeps class proportions only on average. With a rare class, a particular split can be badly off, even leaving a class out of one part. A stratified split fixes the proportion in each part by splitting each class separately and then combining. In scikit-learn, passing stratify=y to train_test_split routes through StratifiedShuffleSplit.

Grouped splits

Often rows are not independent: several visits by one patient, several photos of one object, several sentences from one document. If those rows are split at random, the model sees a patient in training and is tested on that same patient, and it can score well by recognizing the patient instead of the disease. The fix is to split by group: all rows of a group go to one side. GroupShuffleSplit and GroupKFold do this. train_test_split itself has no group argument.

Time-ordered splits

When the task is to predict the future, test data must come after training data. A random split lets the model train on the days surrounding every test day, which no real forecaster ever has. Use a forward split: train on the past, test on the following period. train_test_split(..., shuffle=False) does this once (first rows train, last rows test; the data must already be sorted by time), and TimeSeriesSplit repeats it with a growing training window.

When a random split lies

A random split is honest only if the rows are independent and the future looks like the sample. It misleads when classes are rare, rows come in groups, order matters, near-duplicates exist (the same image resized, the same text reposted), or the deployed data comes from a different place than the collected data. In every case the score looks good and then falls apart in use.

Same data, three ways to cut it. Only the grouped cut keeps patients apart; only the time-ordered cut respects the arrow of time.

train_test_split

train_test_split(*arrays, test_size=None, train_size=None, random_state=None, shuffle=True, stratify=None) returns each input array cut into a train part and a test part, interleaved: X_train, X_test, y_train, y_test. If neither size is given, test_size defaults to 0.25. Sizes are fractions when floats and counts when ints. With shuffle=True it takes one split from ShuffleSplit (or StratifiedShuffleSplit when stratify is given); with shuffle=False it takes the first rows as train and the rest as test, and combining that with stratify raises an error. Set random_state to an integer to make the split reproducible.

Worked example

Implementation

from sklearn.model_selection import (
    train_test_split, GroupShuffleSplit, TimeSeriesSplit,
)

# 1) hold out a test set first, then carve validation out of what remains
X_rest, X_test, y_rest, y_test = train_test_split(
    X, y, test_size=0.15, stratify=y, random_state=0
)
X_train, X_val, y_train, y_val = train_test_split(
    X_rest, y_rest, test_size=0.15 / 0.85, stratify=y_rest, random_state=0
)

# 2) grouped: no patient on both sides
gss = GroupShuffleSplit(n_splits=1, test_size=0.3, random_state=0)
train_idx, test_idx = next(gss.split(X, y, groups=patient_id))

# 3) time-ordered: sorted by time, train on the past
X_train, X_test = X[:-k], X[-k:]            # same as shuffle=False
for train_idx, test_idx in TimeSeriesSplit(n_splits=4).split(X):
    ...                                      # each test fold follows its train fold

Do any preprocessing that learns from data (scaling, imputing, encoding) after splitting and fit it on the training part only; fitting on all rows first lets test information into training.

Tradeoffs

• Data for training vs. data for measuring. Every row held out is a row the model does not learn from. The cost is large for small datasets and negligible for huge ones.
• The validation set wears out. Compare many models on it and the best score is partly luck. A fresh test set, used once, is the only unbiased report.
• Stratifying fixes proportions, not the noise. With very few positives, any split leaves a shaky estimate of the rare class; more data or cross-validation helps, stratification alone does not.
• Grouped and forward splits cost sample size. They give smaller effective test sets and noisier estimates, but those estimates are honest.
• Reproducibility. Without a fixed random_state the split, and so the score, changes from run to run, which can look like a model change. Fix it while developing; vary it to see how much the score moves.
• Never reuse the test set to decide. If you do, relabel it as validation and collect a new test set.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, 2nd ed., section 7.2 and 7.10. Springer, 2009.
• Kaufman, Rosset and Perlich. Leakage in Data Mining: Formulation, Detection, and Avoidance. ACM TKDD, 2012.
• Roberts et al. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 2017.
• scikit-learn 1.9.1: train_test_split, GroupShuffleSplit, TimeSeriesSplit.