Calibration

Problem

Classification metrics like accuracy and AUC judge a model's decisions and its ranking. Many uses need more: the probability itself. A doctor weighs a 3% risk differently from a 30% one; an ad system multiplies click probability by bid; a fraud team sets its review threshold from the expected cost of each case; a language model decides when to say "I'm not sure". All of these read the number as a frequency.

Most models are not trained to make that reading true. A support vector machine outputs a margin, not a probability. Boosted trees and random forests output averages that are systematically pulled away from, or toward, the extremes. A deep network trained with cross-entropy can be highly accurate and still say 99% on cases it gets wrong a quarter of the time. Calibration is the property that the probabilities mean what they say, and the post-processing that restores it without retraining the model.

Intuition

Take every case where a weather forecaster said "70% chance of rain" and count how often it rained. If it rained on about 70% of them, and the same holds for every other number the forecaster uses, the forecaster is calibrated. Calibration says nothing about how useful the forecasts are: someone who says "it rains on 30% of days" every single day is perfectly calibrated and tells you nothing. A good probabilistic model is both calibrated and sharp, or discriminating: its numbers spread out and separate the cases that happen from the ones that don't.

This split suggests the fix. A model that ranks cases well but uses the wrong numbers can be repaired by a monotone function that relabels its outputs: "when you say 95%, that really means 75%". Learning that function needs only held-out predictions and labels, and since it is monotone it leaves the ranking, and so the AUC, as it was.

Mechanics

What calibrated means

A binary classifier with predicted probability p̂ = f(x) is calibrated if, for every value p it outputs,

For K classes the strict version requires the whole predicted vector to match the conditional class frequencies; the version usually measured is confidence calibration, where the probability of the predicted class should equal the accuracy on cases with that confidence. The condition is about a population, never about one prediction: a single 70% forecast is neither right nor wrong.

Reliability diagrams and ECE

Since p̂ is continuous, the condition is checked in bins. Split [0, 1] into B intervals, put each prediction in its bin, and compare each bin's mean predicted probability (its confidence) with the fraction of positives in it (its accuracy, or observed frequency). Plotting accuracy against confidence gives a reliability diagram; a calibrated model lies on the diagonal, and points below it are overconfident on the positive side. scikit-learn's calibration_curve returns exactly these two arrays, with uniform bins by default (n_bins=5) or quantile bins that hold equal numbers of predictions.

The expected calibration error summarizes the diagram as the bin-size-weighted gap, and the maximum calibration error as the worst bin:

ECE is easy to read but depends on the binning: few bins hide errors, many bins leave few cases per bin and add noise, so an ECE should always be reported with its bin scheme. It is a diagnostic, not a training objective; scikit-learn provides the curve but no ECE function.

Proper scoring rules: Brier score and log loss

A proper scoring rule is one whose expected value is optimized by reporting your true belief, so no model can gain by shading its probabilities. Two are standard:

Both reward calibration and sharpness together (the Brier score splits exactly into a reliability term minus a resolution term plus the irreducible uncertainty of the labels, when predictions are grouped by value). The Brier score is bounded and forgiving; log loss punishes confident mistakes without limit, which makes it the one models are trained on and the one temperature scaling minimizes. Neither tells you whether the problem is calibration, which is why they are reported next to a reliability diagram.

Recalibration: Platt, isotonic and temperature

All three methods fit a monotone map g from the model's output to a new probability, on data the model was not trained on. Fitting it on the training set learns nothing useful, since the model's training predictions are more confident than its predictions on new data.

• Platt scaling (Platt, 2000, for SVMs) is a one-feature logistic regression on the model's score s: g(s) = σ(a·s + b). Two parameters, so it needs little data, but it can only produce sigmoid-shaped corrections. To avoid overfitting small sets, Platt replaced the 0/1 labels with targets (N₊ + 1)/(N₊ + 2) and 1/(N₋ + 2), and scikit-learn does the same.
• Isotonic regression (Zadrozny and Elkan, 2002) fits the best non-decreasing step function in squared error. The pool-adjacent-violators algorithm sorts by score, walks left to right, and whenever a block's mean is not below the next block's, merges them into their weighted mean, repeating backwards until the sequence is monotone. It can fix any monotone distortion but has as many parameters as steps, so it overfits small calibration sets; scikit-learn's documentation advises against it well below about a thousand samples.
• Temperature scaling (Guo et al., 2017) divides the logits by one number T > 0 before the softmax: p̂ = softmax(z / T). T is fitted by minimizing log loss on a validation set. T > 1 softens overconfident outputs. One parameter, no change to the argmax, and it works for any number of classes directly; for binary models it is Platt scaling with the intercept fixed at zero.

Why modern networks are overconfident

Guo et al. (2017) found that a 1998 LeNet was well calibrated on CIFAR-100, while a 2016 ResNet, far more accurate, was overconfident: their average confidence exceeds their accuracy. They linked the change to greater depth and width, batch normalization, and training with less weight decay, and observed that test log loss starts to rise while test error keeps falling: the network overfits the loss, not the labels.

The loss explains the incentive. Once a training example is classified correctly, cross-entropy can still be lowered by making its logit larger, without changing any decision. Scaling every logit up shrinks the loss on correct examples toward zero and grows it on wrong ones only linearly, so on a training set the network mostly gets right, growing its logits pays. The result carries over to test data, where more examples are wrong. Temperature scaling undoes exactly this uniform inflation, which is why one parameter fixes so much of it.

Worked example

Implementation

Calibrating a fitted classifier on a held-out set with scikit-learn, and checking it on a third split. Wrapping the model in FrozenEstimator tells CalibratedClassifierCV not to refit it, so ensemble="auto" resolves to False and the calibrator is fitted on all of the calibration data.

from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.frozen import FrozenEstimator
from sklearn.metrics import brier_score_loss, log_loss

model.fit(X_train, y_train)                       # any classifier

calibrated = CalibratedClassifierCV(FrozenEstimator(model), method="temperature")
calibrated.fit(X_cal, y_cal)                      # data the model never saw

for name, clf in [("raw", model), ("calibrated", calibrated)]:
    p = clf.predict_proba(X_test)[:, 1]
    frac_pos, mean_pred = calibration_curve(y_test, p, n_bins=10, strategy="quantile")
    ece = abs(frac_pos - mean_pred).mean()        # equal-count bins, so a plain mean
    print(name, brier_score_loss(y_test, p), log_loss(y_test, p), ece)

With quantile bins every bin holds about the same number of predictions, so the unweighted mean of the gaps is the ECE; with uniform bins it would need the bin counts. In PyTorch, temperature scaling is a single learnable scalar: collect validation logits once, then minimize F.cross_entropy(logits / T, labels) over T and divide by T at inference. Swap "temperature" for "sigmoid" on small calibration sets with a binary model that is biased toward one class, and for "isotonic" only with thousands of calibration examples.

Tradeoffs

• Platt vs isotonic vs temperature. Platt and temperature scaling have one or two parameters and are hard to overfit, but can only apply a sigmoid-shaped correction. Isotonic fits any monotone distortion but needs much more data, produces ties, and can output exact 0 and 1. Temperature scaling is the default for neural networks and multiclass models; Platt's extra intercept helps when the model is biased toward one class.
• Calibration data. Every example spent on calibration is one not spent on training. Cross-validated calibration (CalibratedClassifierCV with cv) reuses the data at the cost of fitting several models.
• ECE vs proper scores. ECE isolates calibration but depends on the bins and can be zero for a useless model; Brier and log loss can't be gamed but mix calibration with discrimination. Report both.
• Average vs subgroup calibration. A model can be calibrated overall and miscalibrated for a subgroup or a slice of the input space. Check reliability diagrams per important segment.
• Shift. Calibration holds for the distribution it was fitted on. When inputs or the base rate change, it degrades first, often before accuracy does, so it needs monitoring and refitting.
• Resampling and class weights. Training on rebalanced data (see class imbalance) deliberately shifts the probabilities toward the minority class; recalibrate on data with the real base rate.

Connections

References

• Guo, Pleiss, Sun and Weinberger. On Calibration of Modern Neural Networks. ICML 2017. arXiv:1706.04599
• Platt. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. In Advances in Large Margin Classifiers, MIT Press, 2000.
• Zadrozny and Elkan. Transforming Classifier Scores into Accurate Multiclass Probability Estimates. KDD 2002.
• Niculescu-Mizil and Caruana. Predicting Good Probabilities with Supervised Learning. ICML 2005.
• Brier. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 1950.
• scikit-learn 1.9.1: CalibratedClassifierCV, calibration_curve, IsotonicRegression, brier_score_loss.