Cross-validation

Problem

A single holdout split of 60 rows leaves a handful of test rows, so the accuracy you measure depends heavily on which rows landed there. Hold out more and you train on less. Cross-validation (CV) splits the data into k parts, trains k times with a different part held out each time, and averages the k scores. Every row is tested exactly once and trained on k − 1 times.

The second problem is using the score to choose. Trying many models or settings and keeping the best CV score selects partly for luck, so the winning score is optimistic. Reporting an honest figure for the whole procedure, selection included, needs the nested form below.

Intuition

Think of k exams, each written by a student who has studied from every chapter except the one on that exam. Averaging the k grades says how well the student handles unseen chapters better than one exam would. How you cut the chapters matters, though: if related questions share a chapter, a student has effectively seen the answer, and if the exams run later than the study material, you must not let a student study from the future. Each splitter below is a rule about what counts as "unseen".

The first n mod k folds get one extra row, so ten rows in three folds is 4, 3, 3.

Mechanics

The splitters

• KFold: contiguous blocks, optionally shuffled first (shuffle=False by default). Plain k-fold assumes rows are independent and identically distributed.
• StratifiedKFold: each fold keeps the class proportions of the whole set. Use it for classification, and above all when a class is rare, since an unlucky fold could otherwise contain none of it. On the labels 0 0 0 0 0 0 1 1 1 1 in two folds, each test fold gets 5 rows with exactly two positives.
• GroupKFold: all rows of a group (a patient, a user, a document) land in the same fold, so no group is on both sides. The test folds on groups a a a b b c c d d e in three folds have sizes 3, 4, 3.
• StratifiedGroupKFold combines both, as well as it can.
• TimeSeriesSplit: the training set is always the past and the test fold the block right after it; the training window grows. With 12 rows and 3 splits the (train, test) sizes are (3, 3), (6, 3), (9, 3). Unlike k-fold, the folds are not a partition: the first block is never tested.
• RepeatedKFold and ShuffleSplit repeat the procedure with different random cuts to average out the effect of any one cut.

How noisy is a CV score?

The k fold scores are not independent (the training sets overlap heavily), so the usual standard-error formula understates the uncertainty. The reliable guide is empirical: the spread of the fold scores shows how much a single fold can mislead, and repeating the CV with different shuffles shows how much the mean depends on where the cuts fell. Larger k gives training sets closer to the full data (less pessimistic bias) but folds that are more alike, and costs k fits. Five or ten is the usual compromise.

Choosing models: nested CV

Pick the best of many settings by CV score and that score is biased upward, because the maximum of many noisy estimates is larger than any one's true value. Nested CV wraps the whole selection in an outer loop: in each outer fold, an inner CV on the outer-training rows chooses the setting, the chosen model is refit on those rows and scored on the untouched outer test fold. The outer average estimates how well the procedure works. In scikit-learn this is a GridSearchCV (inner) passed as the estimator to cross_val_score (outer). The final model is then fit on all data using the same procedure.

Rules that keep it honest

• Put every fitted step (scaling, imputation, feature selection) inside a Pipeline so it is refit within each training fold (see data leakage).
• Split by the unit you will generalize over: patients, not visits; the future, not the past.
• Keep a final test set the CV never touched if the number will be reported.

The outer test fold plays no part in choosing the setting, so its score is not inflated by the choice.

Worked example

Implementation

from sklearn.model_selection import (
    KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit,
    cross_val_score, cross_validate, GridSearchCV,
)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression())

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=cv)           # shape (5,)
print(scores.mean(), scores.std(ddof=1))

result = cross_validate(model, X, y, cv=cv,
                        scoring=["accuracy", "roc_auc"],
                        return_train_score=True)
result["test_roc_auc"]                                  # one per fold

cross_val_score(model, X, y, groups=patient_id, cv=GroupKFold(n_splits=5))
cross_val_score(model, X, y, cv=TimeSeriesSplit(n_splits=5))

# nested: the search is the estimator, the outer CV scores it
search = GridSearchCV(model, {"logisticregression__C": [0.01, 0.1, 1, 10]},
                      cv=KFold(5, shuffle=True, random_state=1))
nested = cross_val_score(search, X, y, cv=KFold(5, shuffle=True, random_state=2))

Passing an integer or None as cv builds the right splitter: stratified k-fold for a classifier with discrete targets, plain k-fold otherwise (via check_cv). Pass groups alongside a group-aware splitter; the splitter ignores it otherwise.

Tradeoffs

• Choice of k. Small k trains on less data (pessimistic estimate) and has more overlap-free noise; large k is closer to unbiased but costs k fits and gives highly correlated folds. Leave-one-out is the extreme and is rarely worth its cost.
• Cost. k fits per candidate, and nested CV is k_outer × k_inner × the number of candidates. Budget for it or reduce the grid.
• Wrong splitter, optimistic score. Plain k-fold on grouped or time-ordered data leaks across the split and overstates performance, often by a lot.
• Shuffling. Shuffle independent rows; never shuffle time series.
• Reusing the CV score. The more choices you make by looking at it, the less it means. A final untouched test set, or nested CV, restores an honest number.
• Large data. With millions of rows a single holdout is already stable and CV is an expense, not a need. For deep networks CV is usually skipped for the same reason.
• The estimate is of a procedure, not a model. The k models differ from the final one trained on everything; the CV score describes how the recipe performs.

Check yourself

Connections

References

• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, chapter 7: model assessment and selection.
• Varma and Simon. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 2006.
• Cawley and Talbot. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. JMLR, 2010.
• Bengio and Grandvalet. No Unbiased Estimator of the Variance of K-Fold Cross-Validation. JMLR, 2004.
• scikit-learn 1.9.1: KFold, cross_validate.