Loss functions

Problem

Training needs a target to minimize. The thing you actually care about, such as accuracy, revenue, or whether a ranking put the right item first, is usually a step function of the model's output: change a weight slightly and it doesn't move, so its gradient is zero almost everywhere and useless for learning. A loss function is a smooth stand-in that is small when predictions are good and has a gradient that points toward better ones.

The choice matters more than it looks. The same data fit with squared error and with absolute error gives different models: one chases outliers, the other ignores them. A classifier trained on squared error over probabilities learns slowly when it is confidently wrong. And a loss averaged over a batch quietly changes meaning when some targets are padding. You need to know what each loss assumes so you can pick the one that matches the problem.

Intuition

Pick a probability model for how targets scatter around your prediction, then score the prediction by how surprised that model is by the real target: the negative log-likelihood. Each noise model gives a familiar loss:

• Gaussian noise → −log p is ½(y − ŷ)²/σ² plus a constant: mean squared error. Its minimizer is the mean.
• Laplace noise → |y − ŷ|/b plus a constant: mean absolute error. Its minimizer is the median.
• Bernoulli outcome with probability σ(z) → binary cross-entropy.
• Categorical outcome with probabilities softmax(z) → cross-entropy.

The shape of a loss tells you how it treats errors. Squared error grows quadratically, so one residual of 10 costs as much as a hundred residuals of 1. Absolute error grows linearly, so large errors get no extra weight. Huber is squared near zero and linear past a threshold δ: smooth where errors are small, robust where they are large.

Squared error punishes large residuals hardest; Huber keeps its smooth bottom but grows only linearly past δ, so outliers can't dominate the gradient.

Mechanics

Regression: squared, absolute, Huber

MSE = (1/N) Σ (yᵢ − ŷᵢ)²    MAE = (1/N) Σ |yᵢ − ŷᵢ|
Huber_δ(r) = ½ r² if |r| ≤ δ, otherwise δ(|r| − ½δ)

The gradients tell the story. Squared error's gradient with respect to ŷ is −2r: it grows with the error, so a big outlier pulls hard. Absolute error's gradient is −sign(r): the same push for every error, which is robust but never shrinks near the answer, so it bounces unless the learning rate decays. Huber's gradient is −r inside δ and −δ·sign(r) outside: proportional when close, clipped when far. PyTorch's SmoothL1Loss is Huber divided by δ (its beta plays the role of δ).

Classification: cross-entropy over logits

A binary classifier outputs a logit z, and p = σ(z) = 1/(1 + e^−z). The Bernoulli negative log-likelihood of a label y ∈ {0, 1} is −[y log p + (1 − y) log(1 − p)]. Computed literally, σ(z) rounds to exactly 1 for large z and the log of 1 − p becomes −∞. Rewriting in terms of z gives a form that never overflows:

BCE(z, y) = max(z, 0) − z·y + log(1 + e^−|z|)

This is what BCEWithLogitsLoss computes, and why it takes logits rather than probabilities. The gradient is simply σ(z) − y. Multi-class is the same idea with softmax: cross-entropy over logits, CE(z, k) = −z_k + log Σ_j e^z_j, with gradient softmax(z) − onehot(k). The entropy page derives it in full.

Margin losses: hinge and logistic

With labels y ∈ {−1, +1}, the margin m = y·z is positive when the sign of z is right and large when it is right with room to spare. The loss you'd like is 0-1: 1 if m ≤ 0, else 0. It has no useful gradient, so it is replaced by a convex upper bound:

• Hinge, max(0, 1 − m): zero once the margin passes 1, so correct and confident points stop contributing. This is the support vector machine's loss.
• Logistic, log(1 + e^−m): binary cross-entropy rewritten in margin form. It never reaches zero, so it keeps nudging every point and gives calibrated probabilities.

Hinge and logistic are convex upper bounds on the 0-1 loss: minimize them and you push the error rate down while still getting gradients.

Ranking and contrastive losses

Sometimes there is no target value, only "this should score higher than that". Ranking losses compare pairs or sets of scores:

• Margin ranking: max(0, −y(s₁ − s₂) + margin), for "s₁ should beat s₂ by at least the margin".
• Triplet: max(0, d(a, p) − d(a, n) + margin), pulling an anchor toward a positive and away from a negative in embedding space.
• InfoNCE (contrastive): cross-entropy where the "classes" are one positive and many negatives, scored by similarity divided by a temperature τ. CLIP and SimCLR use it with every other item in the batch as a negative.

The temperature sets how sharply the loss focuses. A small τ magnifies differences in similarity, so the loss concentrates on the hardest negatives; a large τ spreads attention across all of them.

Reduction and ignore_index

Losses are computed per element, then reduced: reduction="none" keeps the vector, "sum" adds it, and "mean" (the default) divides by the count. For classification, ignore_index marks targets to skip, usually padding tokens. Skipped elements contribute 0 and are also left out of the mean's denominator, so the mean is over real tokens only. With class weights, the mean divides by the sum of the weights of the kept targets instead.

Worked example

The mean moves toward the outlier; the median ignores its size; Huber sits in between, close to the median because only one point lies beyond δ.

Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

y = torch.tensor([2.0, 2.5, 3.0, 3.2, 3.6, 12.0])
c = torch.full_like(y, 3.0)
F.mse_loss(c, y), F.l1_loss(c, y), F.huber_loss(c, y, delta=2.0)

# binary: pass logits, never sigmoid outputs
z = torch.tensor([2.0, 0.0, -3.0, 1000.0])
t = torch.tensor([1.0, 1.0, 1.0, 0.0])
F.binary_cross_entropy_with_logits(z, t, reduction="none")
# tensor([0.1269, 0.6931, 3.0486, 1000.0000])

# multi-class with padding: -100 is the default ignore_index
logits = torch.randn(4, 10)
target = torch.tensor([4, 7, -100, 2])
F.cross_entropy(logits, target)              # mean over the 3 real targets

# embeddings
a, p, n = torch.tensor([[0.0, 0.0]]), torch.tensor([[1.0, 0.0]]), torch.tensor([[1.5, 1.0]])
nn.TripletMarginLoss(margin=1.0)(a, p, n)    # tensor(0.1972)

# InfoNCE: row i of the similarity matrix should pick column i
img, txt = F.normalize(torch.randn(8, 64), dim=-1), F.normalize(torch.randn(8, 64), dim=-1)
tau = 0.07
sim = img @ txt.T / tau
loss = (F.cross_entropy(sim, torch.arange(8)) + F.cross_entropy(sim.T, torch.arange(8))) / 2

The same losses in plain Python:

from math import exp, log1p, log, dist

def huber(r, delta=1.0):
    return 0.5 * r * r if abs(r) <= delta else delta * (abs(r) - 0.5 * delta)

def bce_with_logits(z, y):
    return max(z, 0) - z * y + log1p(exp(-abs(z)))

def hinge(margin):
    return max(0.0, 1 - margin)

def triplet(a, p, n, margin=1.0):
    return max(0.0, dist(a, p) - dist(a, n) + margin)

def info_nce(sims, tau, target=0):
    logits = [s / tau for s in sims]
    top = max(logits)
    return -logits[target] + top + log(sum(exp(x - top) for x in logits))

huber(3, 2), bce_with_logits(1000, 0), triplet((0, 0), (1, 0), (1.5, 1))
# (4.0, 1000.0, 0.197)

Choosing a loss

Tradeoffs

• Robustness versus efficiency. MSE is the most efficient estimator when the noise really is Gaussian, and the worst when it has heavy tails. L1 is robust but its gradient never shrinks, which makes the final steps noisy. Huber needs a δ, which has to match the scale of the targets.
• Units matter. MSE is in squared target units, so rescaling targets rescales the loss and the gradient; normalize targets or adjust the learning rate.
• Surrogates aren't the metric. Lower cross-entropy usually means higher accuracy, but not always; and a loss that ignores class balance can reach a low value by ignoring a rare class. Track the metric you care about separately.
• Reductions change the effective learning rate. Switching from mean to sum multiplies gradients by the batch size. Averaging per sequence and then over sequences weights short sequences' tokens more than averaging over all tokens.
• Contrastive losses depend on the batch. InfoNCE with in-batch negatives gets better with bigger batches, and its temperature is a sensitive hyperparameter (often learned, as in CLIP).

Check yourself

Connections

References

• Huber. Robust Estimation of a Location Parameter. Annals of Mathematical Statistics, 1964.
• Hastie, Tibshirani and Friedman. The Elements of Statistical Learning, 2nd ed., sections 10.6 and 12.3. Springer, 2009.
• Bishop. Pattern Recognition and Machine Learning, sections 1.5 and 4.3. Springer, 2006.
• Cortes and Vapnik. Support-Vector Networks. Machine Learning, 1995.
• Schroff, Kalenichenko and Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. 2015. arXiv:1503.03832
• van den Oord, Li and Vinyals. Representation Learning with Contrastive Predictive Coding. 2018. arXiv:1807.03748
• Lin et al. Focal Loss for Dense Object Detection. 2017. arXiv:1708.02002
• PyTorch v2.15.0: torch/nn/modules/loss.py and torch/nn/functional.py.