Bayes' rule and likelihood

Problem

A test for a rare disease is 90% sensitive and gives a false positive 9% of the time. You test positive. How likely is it that you have the disease? The tempting answer is about 90%. The right answer for this test is under 10%. The data (a positive result) has to be combined with what was believed before it arrived (the disease is rare), and intuition is bad at that combination.

Machine learning faces the same question everywhere. A model has parameters θ and sees data x. The model says how probable the data is for each θ; what we want is the reverse, which θ is plausible given the data. Bayes' rule is the exact way to turn one into the other, and its two simplified uses, maximum likelihood and MAP estimation, are how almost every model in this reference is trained.

Intuition

Think in counts rather than percentages. Out of 10,000 people, 1% (100) are sick, and the test catches 90 of them. Of the 9,900 healthy people, 9% (891) also test positive. So 981 people test positive, and only 90 of them are sick: 9.2%. The false positives win because they come from a group 99 times larger. A positive result multiplies the odds of being sick by 10 (0.9 / 0.09), but from a starting point of 1 in 99.

The highlighted boxes are everyone who tests positive. Bayes' rule is this picture written as a formula.

Mechanics

Bayes' rule

The definition of conditional probability gives the joint probability two ways, P(A, B) = P(A | B) P(B) = P(B | A) P(A). Dividing by P(B):

P(θ | x) = P(x | θ) P(θ) / P(x)     posterior = likelihood × prior / evidence

The prior P(θ) is the belief before the data, the likelihood P(x | θ) says how well each θ explains the data, and the evidence P(x) = Σ_θ P(x | θ) P(θ) just makes the posterior sum to one. For the test, P(positive) = 0.9 × 0.01 + 0.09 × 0.99 = 0.0981, and P(sick | positive) = 0.009 / 0.0981 = 0.0917. Because the evidence does not depend on θ, it is often dropped: posterior ∝ likelihood × prior. In odds form the evidence cancels exactly: posterior odds = likelihood ratio × prior odds.

Today's posterior is tomorrow's prior. A second, independent positive test starts from 0.0917 and ends at 0.5025; a third reaches 0.9099. Each result multiplies the odds by the same factor of 10.

Likelihood versus probability

P(x | θ) is one formula read two ways. With θ fixed and x varying, it is a probability: it sums (or integrates) to one over all possible data. With the data fixed and θ varying, it is the likelihood L(θ) = P(x | θ): a score for each parameter value, which need not sum to one over θ and is not a probability of θ. Flip a coin 10 times and see 7 heads. For a fixed bias θ = 0.7 the probabilities of 0 to 10 heads add to 1; the likelihood of the observed 7 heads, as a function of θ, has area 0.0909 (exactly 1/11). To make a statement about θ itself, you need a prior and Bayes' rule.

For independent examples the likelihood is a product, so in practice everything uses the log: log L(θ) = Σ_i log P(x_i | θ). Products of many small numbers underflow; sums of logs do not. This is why every torch.distributions class exposes log_prob rather than a probability.

Maximum likelihood

Maximum likelihood estimation (MLE) picks the θ that makes the observed data most probable, θ̂ = argmax_θ Σ log P(x_i | θ); equivalently it minimizes the negative log-likelihood (NLL). For the coin, setting the derivative of 7 log θ + 3 log(1 − θ) to zero gives θ̂ = 7/10 = 0.7, the observed frequency.

The common training losses are NLLs of a chosen noise model. If y ∼ Normal(f(x), σ²) with fixed σ, the NLL is Σ(y − f(x))² / 2σ² plus a constant, so least squares is MLE under Gaussian noise; nn.GaussianNLLLoss keeps the variance term for models that predict σ² too. If y ∼ Bernoulli(σ(z)), the NLL is binary cross-entropy, and PyTorch's Bernoulli.log_prob is literally -binary_cross_entropy_with_logits. Categorical labels give cross-entropy (see loss functions).

MAP estimation and priors as regularizers

MLE trusts the data completely. With 0 heads in 3 flips it says θ̂ = 0: the coin can never land heads. Maximum a posteriori (MAP) estimation instead maximizes the posterior, which by Bayes' rule is the likelihood plus a log-prior term:

θ̂_MAP = argmax_θ [ Σ_i log P(x_i | θ) + log P(θ) ]

For the coin, a Beta(a, b) prior acts like a − 1 extra heads and b − 1 extra tails, so θ̂_MAP = (k + a − 1) / (n + a + b − 2). With a Beta(2, 2) prior, 0 heads in 3 gives 0.2 instead of 0: add-one (Laplace) smoothing is MAP estimation.

The log-prior is a regularizer. Take linear regression with Gaussian noise of variance σ² and a Gaussian prior w ∼ Normal(0, τ²) on each weight. The negative log-posterior is, up to constants and a factor 1/2σ²,

‖y − Xw‖² + (σ² / τ²) ‖w‖²

which is exactly ridge regression's objective with α = σ²/τ²: a tighter prior (small τ) means stronger regularization. A Laplace prior gives the L1 penalty of the lasso instead (see regularization). Weight decay in a neural network is the same Gaussian prior. With more data the likelihood term grows with n while the prior stays fixed, so MAP approaches MLE.

The posterior is the product of the other two, renormalized: narrower than either and between their peaks.

Naive Bayes

A classifier can use Bayes' rule directly: model how each class generates features, P(x | y), and the class frequencies, P(y), then predict argmax_y P(y) P(x | y). With many features P(x | y) is impossible to estimate jointly, so naive Bayes assumes the features are independent given the class:

log P(y | x) = log P(y) + Σ_j log P(x_j | y) − log P(x)

The assumption is almost always false (words in an email are not independent), but the classifier still ranks classes well, trains in one pass of counting, and needs little data. Its probabilities tend to be overconfident because correlated features are counted as independent evidence. scikit-learn's estimators all compute the "joint log likelihood" log P(y) + log P(x | y) and normalize it with a log-sum-exp; they differ in the per-feature model: GaussianNB (a Normal per feature and class), MultinomialNB (word counts), BernoulliNB (binary features) and CategoricalNB. MultinomialNB's alpha adds pseudo-counts to every feature: the MAP smoothing above.

In the libraries

Worked example

Implementation

import torch
from torch.distributions import Normal

data = torch.tensor([2.1, 1.7, 2.6, 2.0, 1.4])
mu = torch.zeros((), requires_grad=True)
log_sigma = torch.zeros((), requires_grad=True)
opt = torch.optim.Adam([mu, log_sigma], lr=0.05)
prior = Normal(0.0, 1.0)          # a prior on mu; drop it for plain MLE

for _ in range(500):
    nll = -Normal(mu, log_sigma.exp()).log_prob(data).sum()
    loss = nll - prior.log_prob(mu)   # negative log-posterior: MAP
    opt.zero_grad(); loss.backward(); opt.step()

from sklearn.naive_bayes import MultinomialNB

X = [[3, 2, 0, 1], [1, 0, 3, 4]]   # word counts, one row per class
y = ["spam", "ham"]
clf = MultinomialNB(alpha=1.0).fit(X, y)
clf.predict_proba([[1, 1, 1, 0]])

The MAP loss is the MLE loss plus −log prior, the same shape as a data loss plus a penalty. Parameterizing σ through its log keeps it positive without constraints. The scikit-learn lines pool each class's counts into one row, so the class prior they learn is uniform, not the 2 : 3 of the worked example; pass class_prior=[0.6, 0.4] (classes are sorted, ham first) to match it.

Tradeoffs

• MLE versus MAP. MLE needs no prior but overfits small samples and gives zero probability to anything unseen; MAP fixes that at the price of choosing a prior, whose influence fades as data grows.
• Point estimates versus the full posterior. MLE and MAP return one θ and discard how uncertain it is. Keeping the posterior (Bayesian inference) gives calibrated uncertainty but is rarely tractable for neural networks; ensembles and dropout approximations are the usual compromises.
• MAP depends on the parameterization. The mode of a density moves when θ is transformed (say, to log θ), so MAP is not invariant the way MLE is.
• Naive Bayes. Fast, works with little data and many features, and a strong baseline for text; but the independence assumption makes its probabilities overconfident and it cannot model feature interactions.
• Base rates matter. A classifier's posterior depends on the class prior it was trained with; a model trained on balanced data overstates rare classes when deployed where they are rare.

Connections

References

• Bishop. Pattern Recognition and Machine Learning. Springer, 2006 (Bayes' rule, MLE and MAP, Beta-binomial conjugacy, the Gaussian prior behind ridge).
• Murphy. Probabilistic Machine Learning: An Introduction. MIT Press, 2022 (naive Bayes and smoothing).
• Gigerenzer and Hoffrage. How to Improve Bayesian Reasoning Without Instruction: Frequency Formats. Psychological Review 102(4), 1995 (natural frequencies).
• PyTorch v2.15.0: torch/distributions/distribution.py, normal.py, bernoulli.py.
• scikit-learn 1.9.1: sklearn/naive_bayes.py, _ridge.py.