Monitoring and drift

Problem

Every guarantee from held-out evaluation assumes the test set comes from the same distribution as the data the model will see. After deployment that assumption decays. Customers change, a competitor launches, a sensor is recalibrated, an upstream team renames a field, a pandemic moves everyone online. The model keeps returning numbers with the same confidence, and nothing in the serving path fails: a server answering in 20 ms with stale probabilities looks perfectly healthy.

The labels that would show the problem often come late or never: a loan defaults months after approval, a fraud chargeback arrives weeks after the purchase, a recommendation that wasn't clicked says little about the one that wasn't shown. Monitoring is the practice of catching these changes from whatever signals are available, telling apart the ones that hurt the model from the ones that don't, and deciding when to recalibrate, retrain, or look for a bug.

Intuition

A classifier is an estimate of P(y | x), learned from examples drawn from a joint P(x, y). That joint can change in three ways that matter differently. If only the inputs move, P(x), into regions the model saw rarely, its answers are still right where it learned well and become guesses where it didn't. If only how common each outcome is changes, P(y), every probability is off by the same factor in the odds, and a one-number correction fixes it. If the relationship itself changes, P(y | x), the model is answering a question that no longer has the same answer, and only new labels can say how.

So there are two kinds of signals. Inputs and predictions are available immediately for every request and show that something changed. Outcomes arrive later, for some of the requests, and show whether the model is wrong. A good monitor uses the cheap signals as an early warning and the expensive ones as the verdict.

Mechanics

Three kinds of shift

Write the joint two ways: P(x, y) = P(y | x) P(x) = P(x | y) P(y). Following Moreno-Torres et al. (2012), each kind of shift keeps one factor fixed and changes the other:

• Covariate shift: P(x) changes, P(y | x) does not. A credit model trained mostly on salaried applicants starts seeing freelancers. If the model learned P(y | x) well everywhere, it stays correct; its errors grow only where the new inputs are rare in training. Shimodaira (2000) showed training can be corrected by weighting each example by P_new(x) / P_train(x).
• Label shift (prior shift): P(y) changes, P(x | y) does not. A disease becomes more common, but a sick patient's symptoms look the same. By Bayes' rule the correct posterior moves by a fixed factor in the odds, so the model's probabilities can be corrected without retraining (Saerens et al., 2002).
• Concept drift: P(y | x) changes. Fraudsters adapt to the rules, a feature's meaning changes, a policy change makes the same applicant riskier. The model's mapping is now wrong, and since the inputs may not move at all, nothing short of new labels reveals it.

In time, drift can be sudden (a code release), gradual (tastes changing), recurring (seasons, weekends), or a one-off outlier. Gama et al. (2014) survey the detectors and adaptation strategies for each; models that learn from every new example are covered under online learning.

What to watch

The first line of defence is the plain data checks: the share of nulls, values outside the training range, unseen categories, a feature that is suddenly constant. Most production "drift" is a broken pipeline, not a changing world, and these checks catch it the same day (Breck et al., 2017). Then come distribution tests per feature and on the model's output. The output is the most useful single series: it is one number per request, it summarizes every feature the model uses, weighted by how much the model uses it, and its mean is the base rate the model believes in, which can be compared with the observed rate once labels mature.

The population stability index

Split a feature into bins, usually deciles of the reference (training or launch) data, so each holds 10% of it. Let e_i be the reference share in bin i and a_i the current share. Then

Every term is non-negative, since a_i − e_i and ln(a_i/e_i) always have the same sign. PSI is the sum of the two KL divergences, KL(a ‖ e) + KL(e ‖ a), over the binned distributions, so it is symmetric. The usual reading, a convention from credit scoring with no single canonical source, is below 0.1 stable, 0.1 to 0.25 worth watching, above 0.25 a significant shift. A bin that is empty in either distribution makes its term infinite, so implementations clip shares to a small ε; the result then depends on ε.

The two-sample Kolmogorov–Smirnov test

For a continuous feature, compare the empirical CDFs F_ref and F_cur directly, with no bins:

Under the null hypothesis that both samples come from one distribution, √(nm/(n+m)) · D follows the Kolmogorov distribution for large samples, so the test rejects at level α when D exceeds c(α) · √((n+m)/(nm)) with c(α) = √(−ln(α/2) / 2), 1.358 at α = 0.05. The p-value is the series 2 Σ_k≥1 (−1)^k−1 e^−2k²λ², with λ = (√N_e + 0.12 + 0.11/√N_e) · D and N_e = nm/(n+m) (Stephens' correction). The catch is that the critical value shrinks like 1/√n: with enough traffic every real-world feature "fails" the test, so KS answers "is there any difference?", not "is the difference big enough to matter?". PSI is a size, KS a significance; dashboards usually alert on a size.

Delayed and partial labels

When labels arrive over weeks, the most recent cohorts always look better than they are: only the quick outcomes are in. Comparing this week's observed rate with last quarter's fully matured one finds a false improvement every time. Compare cohorts at the same age, wait for a maturity window, or model the arrival curve. Labels can also be biased by the model itself: a rejected loan never shows whether it would have defaulted, and a fraud model only gets labels for the cases it flagged for review. Those gaps are filled with a small random holdout that is served without the model's decision, which also keeps the evaluation honest for causal questions.

Correcting label shift

If P(x | y) is fixed and the positive rate moves from π to π′, Bayes' rule gives the new posterior from the old one:

In log-odds this adds the constant ln(π′/(1−π′)) − ln(π/(1−π)) to every prediction. π′ itself is unknown without labels, but can be estimated from the predictions on new data: Saerens et al. (2002) iterate EM between the corrected posteriors and their mean, and Lipton et al. (2018) solve a linear system using the model's confusion matrix from validation (black-box shift estimation), which also detects label shift with a statistical test.

Retraining triggers

• Schedule. Retrain daily, weekly or monthly on a sliding window. Simple and predictable, wasteful when nothing changed, too slow after a sudden break.
• Input drift. PSI above the action threshold on an important feature or on the score. Fast, but covariate shift often doesn't hurt, so this trigger should open an investigation, not ship a model.
• Performance. A matured-label metric (Brier, log loss, AUC, calibration gap) crosses a threshold or a drift detector fires. The signal that matters, late by the label delay.
• Calibration only. When ranking holds (AUC unchanged) but the probabilities are off, refit a calibrator on recent labeled data: two parameters, minutes, and no new model to validate.

Whatever fires, the new model goes through the same evaluation, shadow or canary rollout as the first one; an automatic retrain that trains on a broken feed ships the bug.

Worked example

One feature x, standard normal at launch. The true outcome probability is σ(2x − 1), and the deployed model learned it exactly: p̂ = σ(2x − 1). Each week is 2000 evenly spaced quantiles of that week's input distribution, an idealized sample with no noise, and each point's outcome is scored by its true probability, so every number below is the expected value, not one random draw. Monitoring uses the launch deciles of x as PSI bins.

Implementation

A batch monitoring job: PSI and KS per feature against a reference snapshot, and, for the cohort whose labels have matured, calibration metrics and a refit calibrator for the frozen deployed model.

import numpy as np
from scipy.stats import ks_2samp
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.frozen import FrozenEstimator
from sklearn.metrics import brier_score_loss, log_loss, roc_auc_score

def psi(reference, current, bins=10, eps=1e-4):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1)[1:-1])
    e = np.bincount(np.searchsorted(edges, reference, side="right"), minlength=bins) / len(reference)
    a = np.bincount(np.searchsorted(edges, current, side="right"), minlength=bins) / len(current)
    empty = (a == 0) | (e == 0)
    e, a = np.clip(e, eps, None), np.clip(a, eps, None)
    return float(np.sum((a - e) * np.log(a / e))), bool(empty.any())

for name in features:                          # inputs: every request, now
    value, emptied = psi(X_ref[name], X_week[name])
    d, p = ks_2samp(X_ref[name], X_week[name])
    report(name, psi=value, emptied=emptied, ks=d, ks_p=p)

scores = model.predict_proba(X_mature)[:, 1]   # outcomes: the matured cohort
frac_pos, mean_pred = calibration_curve(y_mature, scores, n_bins=5)
report("model", auc=roc_auc_score(y_mature, scores),
       brier=brier_score_loss(y_mature, scores), log_loss=log_loss(y_mature, scores),
       max_gap=np.abs(frac_pos - mean_pred).max(),
       rate_gap=scores.mean() - y_mature.mean())

recalibrated = CalibratedClassifierCV(FrozenEstimator(model), method="sigmoid")
recalibrated.fit(X_mature, y_mature)           # two parameters; the model is not refit

FrozenEstimator makes CalibratedClassifierCV use all the data for the calibrator and leave the model alone. scikit-learn's _sigmoid_calibration parameterizes Platt as σ(−(a·s + b)) and smooths the 0/1 targets, so its a has the opposite sign of the one above. The recalibrated model should itself be scored on a later cohort before it replaces the old one. Thresholds and reference windows belong in configuration next to the model version, since a retrained model needs a new reference snapshot: its own training data.

Tradeoffs

• Sensitivity vs alert fatigue. Hundreds of features times daily tests produce false alarms every day; a team that learns to ignore them misses the real one. Alert on the score and a few key features, use sizes rather than p-values, and require persistence (two windows in a row).
• PSI vs KS vs model-based detectors. PSI is easy to explain and handles categories but depends on the bins and ε. KS needs no bins but is univariate and over-sensitive at volume. Training a classifier to tell reference from current data (a domain classifier) catches multivariate shift that no single feature shows; Rabanser et al. (2019) found that testing a dimension-reduced representation, or the model's own outputs, works well for high-dimensional inputs.
• Reference window. Comparing with training data detects everything since launch; comparing with last week detects sudden changes but lets slow drift through unnoticed. Seasonal data needs last year's same week.
• Label delay vs label quality. Early, partial labels give a fast but biased verdict; waiting for maturity is accurate but slow. Proxy outcomes (a click for a purchase) are faster still and drift on their own.
• Recalibrate vs retrain vs adapt online. Recalibration is cheap and safe but fixes only monotone errors. Retraining fixes concept drift but needs fresh labels, validation and a rollout. Online updates adapt fastest and can also learn a corrupted feed within hours.
• Feedback loops. A deployed model changes the data it later trains on: approved loans are the only labeled loans, recommended items get the clicks. Sculley et al. (2015) list these hidden loops among ML systems' technical debts; a small randomized holdout is the usual remedy.

Connections

References

• Moreno-Torres, Raeder, Alaiz-Rodríguez, Chawla and Herrera. A Unifying View on Dataset Shift in Classification. Pattern Recognition, 2012.
• Shimodaira. Improving Predictive Inference under Covariate Shift by Weighting the Log-Likelihood Function. Journal of Statistical Planning and Inference, 2000.
• Saerens, Latinne and Decaestecker. Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation, 2002.
• Lipton, Wang and Smola. Detecting and Correcting for Label Shift with Black Box Predictors. ICML 2018. arXiv:1802.03916
• Rabanser, Günnemann and Lipton. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. NeurIPS 2019. arXiv:1810.11953
• Gama, Žliobaitė, Bifet, Pechenizkiy and Bouchachia. A Survey on Concept Drift Adaptation. ACM Computing Surveys, 2014.
• Sculley et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS 2015.
• Breck, Cai, Nielsen, Salib and Sculley. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE Big Data 2017.
• Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large Margin Classifiers, MIT Press, 2000.
• scikit-learn 1.9.1: calibration_curve, CalibratedClassifierCV, brier_score_loss, log_loss.