Classification metrics

Problem

Suppose one email in a hundred is a phishing attempt. A filter that labels every email "fine" is right 99% of the time and catches nothing. Accuracy answers "how often was the label right?", which is rarely the question you care about. A hospital wants to know how many sick patients the screen finds and how many healthy ones it frightens; a search engine wants to know how many of the top results are good; a fraud team wants a ranked list it can work down until the budget runs out.

Each of these is a different summary of the same raw material: how the model's predictions line up with the truth, and, for models that output a score, how that alignment changes as you move the cutoff. Choosing the metric is choosing what to optimize, so it deserves the same care as choosing the model.

Intuition

Think of a metal detector at a beach. Turn the sensitivity up and it finds every coin, but it also beeps at bottle caps and you spend your day digging junk. Turn it down and every beep is a real coin, but you walk past coins. Recall is the share of the coins you found; precision is the share of beeps that were coins. Sensitivity is the threshold, and no setting gets both to 100% unless the detector is perfect.

A ROC curve and a precision–recall curve are simply the record of what happens at every sensitivity setting, so you can judge the detector before deciding where to set the dial. The area under a curve compresses that record into one number when you need to compare detectors.

Every threshold metric is a ratio of these four counts.

Mechanics

Counts and ratios

For a binary problem, mark one class positive. A prediction is one of four things: a true positive (TP), false positive (FP), false negative (FN) or true negative (TN).

• Accuracy = (TP + TN) / n. Honest only when the classes are balanced and the two mistakes cost about the same.
• Precision = TP / (TP + FP): of what the model flagged, how much was right. Falls when you tolerate more false alarms.
• Recall (sensitivity, true-positive rate) = TP / (TP + FN): of what was out there, how much did the model find.
• False-positive rate = FP / (FP + TN); specificity = 1 − FPR.
• F1 is the harmonic mean of precision and recall, 2PR / (P + R) = 2TP / (2TP + FP + FN). The harmonic mean is dragged toward the smaller number, so you cannot hide a poor recall behind a high precision. fbeta_score weights recall β times as much as precision.
• When a denominator is zero (no predicted positives, so precision is 0/0), scikit-learn's zero_division parameter decides the value; the default "warn" returns 0 and raises a warning.

Scores, thresholds and curves

Most models output a score or probability, and a label requires a threshold. Sweeping the threshold from high to low traces out a path. The ROC curve plots recall (TPR) against the false-positive rate. A random scorer follows the diagonal; a perfect one hugs the top-left corner. The area under it (AUC) has a clean meaning: the probability that a randomly chosen positive gets a higher score than a randomly chosen negative, counting ties as half. It only depends on the ordering of the scores, so it ignores calibration and any threshold.

The precision–recall curve plots precision against recall. Precision depends on the class balance, which ROC's FPR hides: with 1,000 negatives per positive, a 1% false-positive rate is ten false alarms for every real hit, and the ROC curve still looks excellent. When positives are rare and you care about the flagged list, the PR curve is the more informative one. Average precision summarizes it as AP = Σ (R_n − R_n−1) P_n, the precision at each threshold weighted by the recall gained there. scikit-learn's implementation is a step sum, not the trapezoid area, because linear interpolation between PR points is too optimistic.

Log loss

If the model outputs probabilities, log loss (cross-entropy) scores them directly: −(1/n) Σ [y log p + (1 − y) log(1 − p)]. It is the quantity logistic regression and neural classifiers minimize (see loss functions). A confident wrong answer is punished without bound: predicting 0.01 for something that happens costs 4.6 nats, while 0.99 for something that happens and 0.01 for something that does not together cost only 0.01 nats on average. scikit-learn clips probabilities to [ε, 1 − ε] with ε the machine epsilon of the dtype, so a stated certainty never produces an infinity.

Choosing a threshold

The threshold is a decision, separate from the model. Set it from costs: if a missed fraud costs 100 times a false alarm, flag far more cases than the 0.5 default would. Or set it from capacity: review the top 50 a day, which is a precision-at-k decision. Or set it from a requirement: the highest threshold that still gives 95% recall. Choose it on validation data, never on the test set (see data leakage), and re-check it when class balance shifts.

Many classes: micro versus macro

With more than two classes, compute each class's metric one-vs-rest and then combine. Macro averaging takes the unweighted mean of the per-class scores, so a rare class counts as much as a common one. Micro pools all the TP, FP and FN first and computes one ratio, so common classes dominate (for single-label problems micro-F1 equals accuracy). Weighted averaging weights the per-class scores by support. Report macro when every class matters, micro when each example matters equally.

The same ten examples. Each dot is one threshold; the vertical axes are true-positive rate on the left and precision on the right.

Worked example

Implementation

from sklearn.metrics import (
    confusion_matrix, precision_score, recall_score, f1_score,
    roc_auc_score, roc_curve, average_precision_score,
    precision_recall_curve, log_loss,
)

y_true  = [1, 1, 0, 1, 0, 1, 0, 0, 0, 0]
y_score = [0.95, 0.85, 0.80, 0.65, 0.55, 0.45, 0.40, 0.30, 0.20, 0.10]
y_pred  = [int(s >= 0.5) for s in y_score]

confusion_matrix(y_true, y_pred)          # [[tn, fp], [fn, tp]]
precision_score(y_true, y_pred)           # tp / (tp + fp)
recall_score(y_true, y_pred)              # tp / (tp + fn)
f1_score(y_true, y_pred)                  # 2tp / (2tp + fp + fn)

roc_auc_score(y_true, y_score)            # needs scores, not labels
average_precision_score(y_true, y_score)  # step sum, not the trapezoid
fpr, tpr, thresholds = roc_curve(y_true, y_score)
precision, recall, thresholds = precision_recall_curve(y_true, y_score)

log_loss(y_true, y_score)                 # needs probabilities
f1_score(y_multi, y_pred_multi, average="macro")   # or "micro", "weighted"

Note the row and column order of confusion_matrix: rows are true classes and columns are predicted classes, sorted by label, so for labels 0 and 1 the layout is [[TN, FP], [FN, TP]].

Tradeoffs

• ROC versus PR. ROC is insensitive to class balance, which is both a virtue (comparable across datasets) and a trap (it can look excellent when the flagged list is mostly false alarms). Use PR or precision-at-k when positives are rare and you act on the top of the list.
• Threshold-free versus threshold-based. AUC and AP rank models without a cutoff but say nothing about performance at the cutoff you will deploy. Report a threshold metric at the operating point too.
• F1 treats the two errors alike. If a miss is much worse than a false alarm, use F-beta with β > 1 or, better, a cost-weighted score; F1 also ignores true negatives entirely.
• Log loss versus accuracy. Log loss rewards well-calibrated probabilities and punishes overconfidence, while accuracy and AUC ignore how sure the model was. They can disagree about which of two models is better (see loss functions).
• Variance. With few positives every metric is noisy. A recall measured on 20 positives has a standard error near 0.1; report intervals or cross-validated spreads, not a bare number.
• Averaging choices. Macro, micro and weighted answer different questions and can differ by a factor of two, as above. State which one you report.
• Metric versus loss. Most of these metrics are not differentiable, so models are trained on a surrogate (log loss) and judged on another; the gap between them is a real risk to monitor.

Check yourself

Connections

References

• Fawcett. An Introduction to ROC Analysis. Pattern Recognition Letters, 2006.
• Davis and Goadrich. The Relationship Between Precision-Recall and ROC Curves. ICML, 2006.
• Saito and Rehmsmeier. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE, 2015.
• scikit-learn 1.9.1: f1_score, roc_auc_score, average_precision_score, log_loss.