Entropy, cross-entropy and KL divergence

Problem

A classifier outputs a probability distribution over classes, and a language model outputs one over the next token. To train them, you need a single number that says how good a predicted distribution q is when the truth is p, and it has to behave well under gradient descent. Accuracy is no use: it ignores confidence and has zero gradient almost everywhere. Squared error between probability vectors works, but it barely punishes a confident wrong answer.

You also need a way to compare two distributions in other settings: keeping a fine-tuned model close to its starting point, matching a student to a teacher, or pulling a VAE's latent code toward a prior. And you need to say how much one variable tells you about another. All of these are measured in the same unit, bits (or nats).

Intuition

Learning that an outcome of probability p happened gives you −log₂ p bits of information. A fair coin flip is 1 bit. An event with probability 1/8 is 3 bits: it is like learning the answers to three yes-or-no questions. A certain event is 0 bits; you learned nothing.

Entropy is the average surprise of a distribution, which is also the fewest bits per outcome any code can use to transmit its samples. Cross-entropy is the average surprise when outcomes come from p but you expected q, the bits you'd spend using a code built for q. KL divergence is the difference: the extra bits you pay for believing q when the truth is p. It is zero only when q = p.

Entropy measures uncertainty: zero for a certain outcome, largest when every outcome is equally likely.

Mechanics

Entropy

H(p) = −Σ_x p(x) log p(x) = E_x∼p[−log p(x)]

With log₂ the unit is bits; with the natural log it is nats (1 nat ≈ 1.443 bits). Frameworks use nats. By convention 0 · log 0 = 0, so impossible outcomes contribute nothing. For k outcomes, entropy runs from 0 (one certain outcome) to log k (uniform): a fair die has log₂ 6 ≈ 2.585 bits.

Cross-entropy and KL divergence

H(p, q) = −Σ_x p(x) log q(x)     D_KL(p ‖ q) = Σ_x p(x) log (p(x) / q(x)) = H(p, q) − H(p)

KL divergence is never negative (Gibbs' inequality) and is zero only when p = q, so H(p, q) ≥ H(p): you can't beat the true distribution's code. It is not symmetric, so it isn't a distance. If q gives probability 0 to an outcome p can produce, D_KL(p ‖ q) is infinite: you claimed it was impossible, and it happened.

The asymmetry has practical consequences. Minimizing D_KL(p ‖ q) over q ("forward" KL, what maximum likelihood does) punishes q for missing any region where p has mass, so q spreads out to cover everything. Minimizing D_KL(q ‖ p) ("reverse" KL, what variational inference does) punishes q for putting mass where p has little, so q locks onto one mode.

Same two distributions, two different divergences. The first argument decides whose probabilities weight the log-ratios.

Why cross-entropy is the classification loss

In classification the target p is one-hot: probability 1 on the true class y. Then H(p) = 0, the cross-entropy collapses to a single term, and it equals the KL divergence:

H(p, q) = −log q(y) = D_KL(p ‖ q)

That is the negative log-likelihood of the true label. Averaged over a dataset, minimizing it is maximum likelihood, and it equals minimizing the KL divergence from the data's distribution to the model's. With q = softmax(z) over logits z, the loss and its gradient are

ℓ = logsumexp(z) − z_y     ∂ℓ/∂z = softmax(z) − onehot(y)

The gradient is just "predicted minus target", bounded in [−1, 1] per logit. Computing the loss straight from logits through log-softmax never forms a tiny probability and takes its log, which is why PyTorch's cross_entropy takes logits, not probabilities.

A confident wrong answer is expensive: the loss is unbounded as q(y) → 0, which is what squared error on probabilities fails to do.

Perplexity

PPL = exp( (1/N) Σ_t −ln q(x_t | x_<t) )

A language model's loss is the mean cross-entropy per token, in nats. Exponentiating it gives perplexity: the number of equally likely choices that would be as surprising. A model that spreads probability uniformly over 50,000 tokens has perplexity 50,000; one that always gives the true token probability 1/4 has perplexity 4. It is the geometric mean of 1/q over the tokens.

Mutual information

I(X; Y) = D_KL( p(x, y) ‖ p(x) p(y) ) = H(X) + H(Y) − H(X, Y)

Mutual information is how far a joint distribution is from independence: the bits that knowing Y saves you about X. It is zero exactly when X and Y are independent, and unlike covariance it catches non-linear dependence. It underlies decision-tree information gain and contrastive objectives like InfoNCE.

Worked example

Implementation

import torch
import torch.nn.functional as F
from torch.distributions import Categorical, kl_divergence

logits = torch.tensor([[2.0, 1.0, 0.0]])
F.cross_entropy(logits, torch.tensor([0]))      # tensor(0.4076): -log softmax(logits)[0]
F.cross_entropy(logits, torch.tensor([2]))      # tensor(2.4076)

p = Categorical(probs=torch.tensor([0.5, 0.25, 0.125, 0.125]))
q = Categorical(probs=torch.full((4,), 0.25))
p.entropy() / torch.log(torch.tensor(2.0))      # tensor(1.7500) bits; entropy() is in nats
kl_divergence(p, q) / torch.log(torch.tensor(2.0))   # tensor(0.2500)

# F.kl_div wants log q as input and p as target: KL(target || exp(input))
F.kl_div(q.logits, p.probs, reduction="sum")    # 0.1733 nats = 0.25 bits

# soft targets work too, e.g. label smoothing or distillation
F.cross_entropy(logits, torch.tensor([0]), label_smoothing=0.1)

The same quantities in plain Python, in bits:

from math import log2, inf

def entropy(p):
    return sum(-pi * log2(pi) for pi in p if pi > 0)

def cross_entropy(p, q):
    if any(pi > 0 and qi == 0 for pi, qi in zip(p, q)):
        return inf
    return sum(-pi * log2(qi) for pi, qi in zip(p, q) if pi > 0)

def kl(p, q):
    return cross_entropy(p, q) - entropy(p)

p, q = [0.5, 0.25, 0.125, 0.125], [0.25] * 4
entropy(p), cross_entropy(p, q), kl(p, q)        # (1.75, 2.0, 0.25)
kl([0.9, 0.1], [0.5, 0.5]), kl([0.5, 0.5], [0.9, 0.1])   # (0.531, 0.737)

Tradeoffs

• Always pass logits. Applying softmax and then log loses precision when a probability underflows to 0 and gives −inf. F.cross_entropy fuses log-softmax and the negative log-likelihood; don't put a softmax layer before it.
• Overconfidence. Cross-entropy keeps rewarding larger logits on correct answers even when accuracy is perfect, which produces overconfident, poorly calibrated models. Label smoothing and temperature scaling counter it.
• Forward or reverse KL. Forward KL (maximum likelihood) covers every mode and can put mass in implausible places; reverse KL picks one mode and can ignore others. Choosing the direction is a modelling decision, not a detail.
• Infinite KL. Any zero in q where p is positive makes KL infinite and its gradient useless. Smooth or mix in a little uniform mass when comparing empirical histograms.
• Perplexities aren't comparable across tokenizers. A model with a different vocabulary splits text into a different number of tokens, so compare bits per byte or per character instead.

Check yourself

Connections

References

• Shannon. A Mathematical Theory of Communication. Bell System Technical Journal, 1948.
• Cover and Thomas. Elements of Information Theory, 2nd ed., chapter 2. Wiley, 2006.
• MacKay. Information Theory, Inference, and Learning Algorithms, chapters 2 and 4. Cambridge, 2003.
• Goodfellow, Bengio and Courville. Deep Learning, section 3.13. MIT Press, 2016.
• Müller, Kornblith and Hinton. When Does Label Smoothing Help? 2019. arXiv:1906.02629
• PyTorch v2.15.0: torch/distributions/kl.py, F.cross_entropy and CrossEntropyLoss.