Class imbalance

Problem

Fraud, disease, machine failure, a click on an ad: the cases you most want to find are usually rare. Take a dataset of 1,000 transactions with 10 frauds. A model that answers "not fraud" every time is right on 99% of them, and it finds none of the frauds. Its recall is 0.

The training loss has the same blind spot. Cross-entropy averages over examples, so the 990 easy negatives outvote the 10 positives, and the cheapest way to lower the loss is to push every prediction toward "negative". The model can still rank the frauds above most legitimate transactions, yet at the default 0.5 cutoff it flags few of them. The data is not wrong. The loss and the threshold were set up for a balanced world.

Intuition

Picture a jury of 1,000 voices where 990 always say "innocent". To hear the other 10, you can make each of them louder (class weights), invite more of them or fewer of the others (oversampling or undersampling), or lower the bar for a "guilty" verdict (threshold moving). A fourth option quiets the voices that are already sure of themselves, so the uncertain ones carry the decision (focal loss).

The first three do nearly the same thing to a well-specified model: they shift its decision boundary toward the majority class. They differ in what they cost and in what happens to the predicted probabilities. Whatever you choose, judge it with metrics that look at the rare class directly: recall, precision, the precision–recall curve, balanced accuracy.

The "balanced" heuristic gives each class the same total say in the loss.

Mechanics

Why accuracy misleads

Accuracy is (TP + TN) / n. With positive rate π, the constant "negative" classifier scores 1 − π, so every accuracy number has to be read against that floor. Better summaries for a rare positive class (see classification metrics):

• Recall and precision of the rare class, and F1 or F_β to combine them with a stated preference.
• Balanced accuracy, the mean of per-class recalls. The constant classifier scores 0.5 however rare the class is. scikit-learn's version equals accuracy with class-balanced sample weights.
• Average precision (area under the precision–recall curve). Its chance level is π, not 0.5, so it shows a rare-class ranker's quality where ROC AUC can look excellent while precision is poor.
• Per-class confusion matrices and cost-weighted error, when you know what each mistake costs.

Class weights

Give each example a weight s_i that depends on its class and minimize the weighted loss:

L(θ) = Σ_i s_yi · ℓ(f_θ(x_i), y_i)     balanced: s_c = n / (K · n_c)

With K classes and n_c examples in class c, the "balanced" weights make every class's weights sum to n/K. scikit-learn exposes this as class_weight="balanced" on most classifiers and multiplies it with any sample_weight. PyTorch's CrossEntropyLoss(weight=…) takes one weight per class; with reduction="mean" it divides by the sum of the weights of the batch's targets, not by the batch size. BCEWithLogitsLoss(pos_weight=p) multiplies only the positive term, −[p·y log σ(x) + (1 − y) log(1 − σ(x))], and its docstring suggests p = negatives / positives. XGBoost's scale_pos_weight is the same idea for boosted trees, with the same suggested value.

Resampling

• Random oversampling repeats minority examples. Repeating each positive k times gives exactly the same loss as weighting it by k, so for a loss-minimizing model the two coincide. Duplicates do not add information, and models that can memorize (deep trees, large networks) overfit them.
• Random undersampling drops majority examples. Training is cheaper and the boundary moves the same way, but information is thrown away; repeating it over several subsamples and averaging (an ensemble) recovers some of it.
• SMOTE creates synthetic positives by interpolating between a positive and one of its positive nearest neighbours. It needs meaningful distances, so it suits low-dimensional numeric features better than text or images. It is not in scikit-learn itself.
• Sampling in a data loader. PyTorch's WeightedRandomSampler draws indices with probability proportional to per-sample weights (by default with replacement). Weighting each example by 1/n_c makes each class equally likely per draw: here a positive share of 50%.

Resample only the training split, inside each cross-validation fold. Resampling before the split puts copies of the same example on both sides, which is leakage.

Threshold moving and costs

A probabilistic classifier separates two decisions: estimating p = P(y = 1 | x), and acting on it. If a false positive costs c_FP and a false negative c_FN, predicting positive is cheaper in expectation when (1 − p)·c_FP < p·c_FN, that is when

p > t* = c_FP / (c_FP + c_FN)

The default 0.5 assumes equal costs. Weighting positives by w is the same as multiplying their cost by w; for a logistic model it adds roughly log w to the logit, which is equivalent to keeping the unweighted model and moving the threshold to 1/(1 + w). scikit-learn's FixedThresholdClassifier sets a cutoff by hand, and TunedThresholdClassifierCV picks the one that maximizes a metric (balanced accuracy by default) across cross-validation folds. Threshold moving keeps the probabilities calibrated; weighting and resampling do not.

Correcting probabilities after resampling

Keeping a fraction β of the negatives multiplies the odds of the positive class by 1/β, by Bayes' rule: the likelihoods are unchanged and only the prior ratio moves. Undo it by multiplying the odds back by β:

odds(p) = β · odds(p_s)   ⇒   p = β p_s / (β p_s − p_s + 1)

The same correction with β = s₀/s₁ applies to a class-weighted model. Do this when downstream code needs real probabilities: expected cost, a risk score shown to a person, or a calibration check.

Focal loss

Lin et al. (2017) proposed focal loss for dense object detection, where a detector scores around 10⁵ candidate boxes per image and almost all are easy background. Writing p_t for the probability the model gives the true class,

FL(p_t) = −α_t (1 − p_t)^γ log p_t

With γ = 0 and α_t = 1 it is cross-entropy. The factor (1 − p_t)^γ shrinks the loss of examples the model already gets right, whatever their class, so the gradient concentrates on hard examples. α_t is an ordinary class weight. The paper's best setting was γ = 2 with α = 0.25 for the rare class, lower than you might expect, because the focusing term already down-weights the many easy negatives. PyTorch has no focal-loss module at v2.15.0; it is a few lines on top of binary_cross_entropy_with_logits (below).

Raising γ leaves hard examples (left) almost untouched and silences easy ones (right).

In the libraries

Worked example

Weighting shifts the curve left by about log 9 / w; moving the threshold reads the old curve lower. Both cut near x = 1.

Implementation

import torch
import torch.nn.functional as F
from torch.utils.data import DataLoader, WeightedRandomSampler

counts = torch.bincount(train_labels)                 # e.g. tensor([990, 10])
class_w = counts.sum() / (len(counts) * counts)       # sklearn's "balanced"
loss_fn = torch.nn.CrossEntropyLoss(weight=class_w.float())
bce = torch.nn.BCEWithLogitsLoss(pos_weight=counts[0] / counts[1])

sample_w = (1.0 / counts.float())[train_labels]       # or: rebalance batches instead
loader = DataLoader(train_ds, batch_size=64,
                    sampler=WeightedRandomSampler(sample_w, len(sample_w)))

def focal_loss(logits, targets, gamma=2.0, alpha=0.25):
    ce = F.binary_cross_entropy_with_logits(logits, targets, reduction="none")
    p_t = torch.exp(-ce)                                # probability of the true class
    alpha_t = alpha * targets + (1 - alpha) * (1 - targets)
    return (alpha_t * (1 - p_t) ** gamma * ce).mean()

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TunedThresholdClassifierCV

clf = LogisticRegression(class_weight="balanced").fit(X_train, y_train)
tuned = TunedThresholdClassifierCV(LogisticRegression(), scoring="f1").fit(X_train, y_train)
tuned.best_threshold_

Use one remedy at a time: a weighted loss on a rebalanced loader counts the correction twice. exp(−ce) recovers p_t from the per-example BCE, which is −log p_t, without a separate sigmoid. The focal loss follows Lin et al.'s equation; alpha=0.25 and gamma=2.0 are the paper's chosen values, not a library default.

Tradeoffs

• Do you need a fix at all? A model that ranks well (high average precision) only needs its threshold set for your costs. Strong learners such as gradient boosting often rank well on imbalanced data without reweighting.
• Threshold moving is cheap, needs no retraining and keeps probabilities calibrated; tune it on validation data. It cannot help a model that never learned to separate the classes.
• Class weights change what the model learns, which matters when capacity is limited or regularization would otherwise flatten the minority. They distort probabilities (correct as above) and make the loss noisier: a few heavily weighted examples dominate each batch.
• Oversampling equals weighting for a fixed loss but costs more compute and invites memorization; undersampling is cheap and throws data away. SMOTE adds synthetic points that may be implausible and needs a sensible distance.
• Focal loss targets the "many easy negatives" case of dense detection; on ordinary tabular imbalance plain weighting is usually as good and simpler. It also miscalibrates probabilities.
• Evaluation pitfalls. Never resample the validation or test set; report metrics at the true base rate. Use stratified splits so every fold has positives (cross-validation). With very few positives, the metric itself is noisy: 20 positives move recall in steps of 0.05.

Connections

The weighted loss is the cross-entropy with per-class multipliers; threshold moving is Bayes' decision rule from Bayes' rule with costs; the probability correction is the same prior-ratio update.

References

• King and Zeng. Logistic Regression in Rare Events Data. Political Analysis 9(2), 2001 (cited by scikit-learn for the "balanced" heuristic). doi:10.1093/oxfordjournals.pan.a004868
• Lin, Goyal, Girshick, He and Dollár. Focal Loss for Dense Object Detection. ICCV 2017. arXiv:1708.02002
• Chawla, Bowyer, Hall and Kegelmeyer. SMOTE: Synthetic Minority Over-sampling Technique. JAIR 16, 2002. arXiv:1106.1813
• Elkan. The Foundations of Cost-Sensitive Learning. IJCAI 2001 (the cost threshold and rebalancing by prior ratio).
• Brodersen, Ong, Stephan and Buhmann. The Balanced Accuracy and Its Posterior Distribution. ICPR 2010 (cited by scikit-learn's balanced_accuracy_score).
• scikit-learn 1.9.1: sklearn/utils/class_weight.py, _classification_threshold.py, balanced_accuracy_score.
• PyTorch v2.15.0: torch/nn/modules/loss.py, torch/utils/data/sampler.py.
• XGBoost v3.4.2: doc/parameter.rst.