Data leakage

Problem

A model is only useful if it works on rows it has not seen, at a moment when the answer is not yet known. We estimate that by holding rows back and scoring on them. Leakage breaks the estimate: something about the held-back rows, or about the future, or about the answer itself, got into the model. The score then measures how well the model uses a shortcut, not how well it predicts.

The danger is that leakage rarely fails loudly. Nothing crashes; the validation number is simply too good. The worse a leak is, the more the result looks like a breakthrough. A very high score on a hard problem is the first thing to distrust.

Intuition

Imagine a student sitting an exam. If they saw the questions beforehand, the mark tells you nothing about what they know. There are three ways that can happen. They may have been handed a column that is secretly the answer key (target leakage). They may have seen the exam paper itself while studying, because the paper and the practice sheets overlapped (train–test contamination). Or they may have read tomorrow's newspaper to predict tomorrow's news (temporal leakage).

The test for all three is the same question: at the moment this prediction would be made in real use, would I have this piece of information? If not, it cannot be in the training recipe, the features or the evaluation, whether it arrived through a column, a row or a statistic.

Different doors, one problem: the model sees something it would not have in production.

Mechanics

Target leakage

A feature is available in your table but is created by, or only exists because of, the outcome. Predicting loan default with a column "number of late-payment notices sent" is the classic case: notices are only sent after someone has missed payments. Others: "discharge medication" in a model for diagnosing the condition; "refund issued" when predicting churn; a column that is a transformation of the label (a rescaled copy, a category that the label defines). Tell-tale signs are a single feature that alone predicts nearly perfectly, and importance that is wildly concentrated.

Train–test contamination

Information flows from evaluation rows into training without a feature naming the label:

• Preprocessing fit on everything. Computing a mean, a scale, a vocabulary, an imputation value, a PCA basis or a target encoding on all rows before the split lets the test rows influence what the model is given.
• Selection fit on everything. Choosing the "best" features, hyperparameters or model by looking at the test score makes the test score a training objective.
• Duplicates and near-duplicates. The same record, image, document or user appears on both sides. The model gets credit for remembering it.
• Groups split apart. Several rows from one patient, device or session are spread over both sides, so the model can recognize the entity, not the pattern. This is why splitters such as GroupKFold exist.

Temporal leakage

When rows are ordered in time, a random split lets the model train on the future and be scored on the past. Even features can leak time: "total purchases this year" computed at year end contains purchases that had not happened on the prediction date. The fix is to split by time (train strictly before validation strictly before test) and to build every feature using only data stamped before the prediction time, including the aggregates. TimeSeriesSplit produces such splits.

The structural fix: one pipeline

Most contamination comes from the order of operations. The safe order is: split first; then, for each training set, fit every learned step (scaler, imputer, selector, encoder, model) on those rows only; finally transform the held-out rows with the fitted steps. A Pipeline enforces that order: its fit clones each transformer, calls fit_transform on the training data one step at a time, then fits the final estimator, and its predict only calls transform on new rows. Cross-validating the whole pipeline refits all of it inside each fold, so no fold's validation rows ever influence a fitted step.

Splitting first costs nothing; fitting before splitting can silently inflate every number that follows.

Worked example

Implementation

from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, TimeSeriesSplit, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# wrong: the selector and scaler have already seen every row
X_sel = SelectKBest(f_classif, k=10).fit_transform(X, y)
cross_val_score(LogisticRegression(), X_sel, y, cv=5)

# right: every learned step is refit inside each training fold
model = Pipeline([
    ("scale", StandardScaler()),
    ("select", SelectKBest(f_classif, k=10)),
    ("clf", LogisticRegression()),
])
cross_val_score(model, X, y, cv=5)

# grouped rows: an entity never straddles the split
cross_val_score(model, X, y, cv=GroupKFold(n_splits=5), groups=patient_id)

# time-ordered rows: train on the past, validate on the future
cross_val_score(model, X, y, cv=TimeSeriesSplit(n_splits=5))

A leakage checklist

• For every feature, can you say when it is known? If it is known only after the outcome, remove it.
• Is the split done before any statistic, selection, imputation, encoding or resampling is fit? Is the whole recipe inside a Pipeline?
• Are there duplicate or near-duplicate records, or several rows per entity, spread across the splits? Split by entity.
• Is the data ordered in time? Split by time and build aggregates with a cutoff.
• Was the test set used once, at the end? Tuning on it makes it a validation set.
• Is the score suspiciously good, or is one feature carrying nearly all the importance? Treat that as a bug until shown otherwise.
• Does the deployed score on fresh data match the offline estimate? A large gap is the final symptom.

Tradeoffs

• Pipelines versus convenience. A pipeline refits every step per fold, which costs time for expensive preprocessing (a large PCA, a heavy encoder). The cost buys an honest estimate; caching with memory reduces it.
• Unsupervised preprocessing on all data. Fitting a scaler or vocabulary on unlabeled test inputs is sometimes done on purpose (transductive learning), but then the reported number is for that setting, not for new data. Say so, or avoid it.
• Gap between train and validation. With time series, features such as rolling windows reach back; leave a gap equal to the longest lookback so windows do not straddle the boundary.
• Near-duplicates are hard to define. Exact-match checks catch the easy cases; images, paraphrased text and the same user with different ids need hashing, embeddings or domain keys. Over-aggressive deduplication discards real data.
• Leakage in the loop. Repeated peeking at one test set (many experiments, one benchmark) leaks slowly through your own decisions. Keep a final untouched set, and expect public leaderboards to be optimistic.
• Pretraining contamination. A large language model may have seen a benchmark's test items in its training corpus. The same logic applies, and the evidence is usually indirect.

Check yourself

Connections

References

• Kaufman, Rosset, Perlich and Stitelman. Leakage in Data Mining: Formulation, Detection, and Avoidance. ACM Transactions on Knowledge Discovery from Data, 2012.
• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, 2nd ed., section 7.10.2 ("The wrong and right way to do cross-validation"). Springer, 2009.
• Kapoor and Narayanan. Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns, 2023.
• scikit-learn 1.9.1: Pipeline, GroupKFold, TimeSeriesSplit, train_test_split.