Missing data

Problem

Most learners need a number in every cell. A linear model cannot multiply a weight by "unknown", and a distance cannot be computed to a point with no coordinate. So something has to happen to each missing value before training: drop the row, drop the column, guess a value, or use a model that has its own rule for missing input.

Every choice changes the data the model sees. Dropping rows can throw away most of a table and, worse, keep a biased subset. Filling with the column mean keeps every row but shrinks the column's spread and its correlation with everything else. And when the fact that a value is missing carries information (people with high incomes skip the income question more often), every filling method erases that signal unless you keep it on purpose.

Intuition

Ask one question first: would the missing values, if you could see them, look like the ones you can see? If a value went missing by pure chance, the observed rows are a fair sample and almost any method is unbiased. If it went missing because of something else you did observe (older people skip a question more), the observed rows are skewed, but you can correct for it using that other column. If it went missing because of its own value (high earners hide their income), the observed rows are skewed in a way the data cannot reveal.

Rubin named these three cases, and his labels are standard. Write M for the 0/1 indicator "this value is missing", Y for the value itself and X for the other, observed columns.

The mechanism is a statement about what M depends on. You can test some of the MAR arrow (does M vary with X?) but never the MNAR arrow, because Y is exactly what is missing.

Mechanics

MCAR, MAR and MNAR

• MCAR, missing completely at random. P(M = 1) does not depend on X or Y. A lab sample dropped on the floor. The complete rows are a random subset, so deleting incomplete rows loses precision but adds no bias.
• MAR, missing at random. P(M = 1) depends only on observed columns. Older patients skip a survey item more, and age is recorded. The complete rows are biased, but within each value of X they are still representative, so a method that conditions on X (regression imputation, iterative imputation, a model that uses X) can undo the bias.
• MNAR, missing not at random. P(M = 1) depends on the missing value itself, even after accounting for X. High earners leave income blank. No method that only looks at the observed data removes the bias without an extra assumption about how M depends on Y; the honest options are to model that assumption explicitly, collect more data, or report how sensitive the result is.

The names are confusing ("at random" in MAR does not mean random); remember them by what M depends on: nothing, observed values, or unobserved values.

Deletion

Listwise deletion (complete-case analysis) drops every row with any missing value. It is unbiased under MCAR and simple, but the losses compound: with 10 columns each missing 10% of values independently, only 0.9¹⁰ ≈ 35% of rows survive. Dropping a column that is mostly empty is often better than dropping the rows. Pairwise deletion uses whatever rows are complete for each pair of columns, which keeps more data but can produce a covariance matrix that no single dataset could have. Deletion also does not help at prediction time: a deployed model still gets rows with holes.

Mean, median and constant imputation

Univariate imputation fills each column with one number learned from that column's observed values: the mean, the median (robust to outliers), the most frequent value (for categories), or a constant you choose. In scikit-learn this is SimpleImputer; fit stores the per-column value in statistics_ and transform writes it into every missing cell. A column with no observed values at all is dropped unless keep_empty_features=True.

It is fast and keeps every row, but it puts a spike of identical values at the center of the column. The column's variance shrinks, its correlation with other columns is pulled toward zero, and any model fit on it becomes overconfident because the filled cells look like real, certain observations. A constant outside the normal range (−1 for a count, "missing" for a category) avoids the false center and lets trees isolate the filled rows, at the cost of a value that linear models treat as real.

Iterative imputation

Multivariate imputation predicts each missing cell from the other columns. scikit-learn's IterativeImputer, modeled on the MICE procedure, does it round-robin:

1. Fill every hole with a simple imputation (the column mean by default).
2. For each column with missing values, in order of increasing missing fraction by default: fit a regressor (default BayesianRidge) on the rows where that column is observed, using the current filled values of all other columns as features, then overwrite that column's missing cells with its predictions.
3. After a full pass, compare with the previous pass. Stop when the largest change in any cell is below tol times the largest absolute observed value, or after max_iter passes (default 10).

It uses correlations between columns, so under MAR it can recover much of what mean imputation throws away. The estimator is still marked experimental and must be enabled with from sklearn.experimental import enable_iterative_imputer. With sample_posterior=True it draws from the regressor's predictive distribution instead of using the mean, which is how you build several imputed datasets for multiple imputation. KNNImputer is the nearest-neighbor alternative: average the column over the k most similar rows.

Missing-indicator features

If missingness itself predicts the target, keep it as a feature. For each column with holes, add a 0/1 column that is 1 where the value was missing, then impute the original column as usual. A linear model can now learn a separate offset for "was missing", and a tree can split on it directly. In scikit-learn, MissingIndicator builds these columns and add_indicator=True on any imputer appends them to its output. By default only columns that had missing values during fit get an indicator.

How gradient-boosted trees route missing values

A tree split asks "is x < t?" and needs an answer for every row. XGBoost and scikit-learn's histogram gradient boosting do not fill missing values. Instead, each split learns a default direction: the side that missing rows go to.

The split search already scans the sorted non-missing values, keeping running sums of gradients G and hessians H on each side. The missing rows' sums are whatever is left over: G_missing = G_node − ΣG_non-missing. XGBoost's histogram evaluator scans twice:

• A forward scan accumulates bins from the left, so the right side is the node total minus the left. Missing rows land on the right.
• A backward scan accumulates from the right, so the left side is the remainder and missing rows land on the left. It runs only when the node actually has missing values.

The candidate with the larger gain wins, and the split records its direction (XGBoost stores it in the top bit of the split index). On a tie the forward scan's candidate stays, so missing goes right. If no missing values were seen at a node during training, only the forward scan runs and missing values at prediction time go right. scikit-learn's HistGradientBoosting follows the same two-scan scheme (its code cites algorithm 3 of the XGBoost paper), but where no missing values were seen during training it sends them to the child with more samples.

Squared loss from a prediction of 0, so each row's gradient is −y and its hessian is 1; λ = 1.0. The missing rows' targets look like the low-x group's, and the backward scan finds that.

Worked example

Implementation

Put the imputer inside a Pipeline so its statistics are learned from the training folds only (see data leakage), and add indicators when missingness might matter:

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import IterativeImputer
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier

simple = make_pipeline(
    SimpleImputer(strategy="median", add_indicator=True),
    LogisticRegression(),
)
iterative = make_pipeline(
    IterativeImputer(max_iter=10, tol=1e-3, random_state=0),
    LogisticRegression(),
)
native = HistGradientBoostingClassifier()   # NaN allowed, no imputer
# XGBoost: xgboost.XGBClassifier(missing=np.nan) learns a default direction per split

The split scan with a default direction, from scratch (the same rule as the worked example):

def score(g, h, lam):
    return g * g / (h + lam)

def best_split(xs, grads, hess, lam=1.0):
    seen = sorted((x, g, h) for x, g, h in zip(xs, grads, hess) if x is not None)
    G, H = sum(grads), sum(hess)
    Gm = G - sum(g for _, g, _ in seen)          # missing rows' sums
    Hm = H - sum(h for _, _, h in seen)
    parent, best = score(G, H, lam), None
    for i in range(1, len(seen)):
        if seen[i][0] == seen[i - 1][0]:
            continue
        gl = sum(g for _, g, _ in seen[:i])
        hl = sum(h for _, _, h in seen[:i])
        sides = [("right", gl, hl)]               # forward scan
        if Hm > 0:
            sides.append(("left", gl + Gm, hl + Hm))  # backward scan
        for go, lg, lh in sides:
            gain = score(lg, lh, lam) + score(G - lg, H - lh, lam) - parent
            if best is None or gain > best[0]:
                best = (gain, (seen[i - 1][0] + seen[i][0]) / 2, go)
    return best

This visits candidates in a slightly different order from XGBoost (which finishes the whole forward scan before the backward one), so on an exact tie between two different thresholds it can pick another one; the tested helper missing_split_scan follows XGBoost's order.

Tradeoffs

• Deletion or imputation? Deletion is defensible under MCAR with few missing rows. Otherwise it wastes data and, under MAR, biases the result. It also gives no answer for rows with holes at prediction time.
• Mean or median? The median is robust to skew and outliers; the mean is right for roughly symmetric columns. Both shrink variance and correlations, which matters most for linear models and for any analysis that reports uncertainty.
• Simple or iterative? Iterative imputation uses the other columns and does much better when columns are correlated, at the cost of one regression per column per pass, a dependence on the chosen regressor, and an estimator that scikit-learn still marks experimental.
• Single or multiple imputation? One filled dataset treats guesses as facts, so standard errors come out too small. Multiple imputation (several draws with sample_posterior=True, results pooled) is the standard fix when you need valid uncertainty, not just a good prediction.
• Indicators. Cheap and often helpful for prediction, but they let the model lean on how data was collected. If the collection process changes (a form field becomes mandatory), the indicator's meaning shifts under the model.
• Native handling. Gradient-boosted trees learn a direction per split, which is usually better than a blind fill and needs no preprocessing. The direction is fixed per split, so missing rows can never be split among themselves, and a feature with no missing values in training gets an arbitrary rule (right in XGBoost, the larger child in scikit-learn).
• No method fixes MNAR from the data alone. Model the mechanism explicitly, collect the reason for missingness, or run a sensitivity analysis over plausible shifts.

Check yourself

Connections

References

• Rubin. Inference and Missing Data. Biometrika, 1976 (the MCAR/MAR/MNAR framework).
• Little and Rubin. Statistical Analysis with Missing Data, 3rd ed. Wiley, 2019.
• van Buuren and Groothuis-Oudshoorn. mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software, 2011.
• Chen and Guestrin. XGBoost: A Scalable Tree Boosting System. KDD 2016, section 3.4 (sparsity-aware split finding).
• scikit-learn 1.9.1: SimpleImputer, MissingIndicator, IterativeImputer, HistGradientBoosting's rule for features with no missing values, sample_goes_left.
• XGBoost v3.4.2: SplitContainsMissingValues, EnumerateSplit.