Preference optimization

Problem

Supervised fine-tuning imitates demonstrations: it raises the probability of whatever a human wrote. That works for format and tone, but it cannot express "this answer is better than that one", it treats every token of a demonstration as equally right, and good demonstrations are slow to write. Judging is cheaper than writing: a person can compare two answers in seconds, even for tasks they could not have answered well themselves.

Preference optimization turns comparisons into a training signal. The data is a set of triples (prompt x, preferred response y_w, rejected response y_l). The goal is a policy π_θ that produces responses people prefer, without drifting so far from the fine-tuned model that it forgets how to write or finds a loophole in the signal.

Intuition

Think of a judge at a cooking contest who never gives scores, only says which of two dishes is better. From enough of those verdicts you can recover a score for every dish: a dish that beats strong dishes must be strong. That is the reward model. Then you train the cook to make dishes the score likes, while keeping them close to the cook's own style, so the cook improves instead of inventing something the judge has never tasted and the score cannot rate.

RLHF does those two steps literally: fit a score, then run reinforcement learning against it with a leash. DPO notices that the best leashed policy has a formula, and that the formula can be turned around so that the policy itself is the score. One classification-style loss on the pairs then does both steps at once.

Both routes start from the same supervised model, which also serves as the frozen reference π_ref. RLHF learns a separate reward and then samples from the policy to optimize it; DPO fits the policy to the pairs directly. The worked example below shows the two reaching the same policy.

Mechanics

Reward models from pairwise preferences

The Bradley–Terry model says each item has a hidden strength, and the chance that i beats j depends only on the difference. Written with a reward r instead of a strength, the probability is the sigmoid of the reward gap:

P(y_w ≻ y_l | x) = σ(r(x, y_w) − r(x, y_l)) = exp r_w / (exp r_w + exp r_l)

A reward model r_φ is usually the fine-tuned language model with its vocabulary head replaced by a single linear output, read at the last token of the response. It is trained by maximum likelihood on the comparisons, which is binary cross-entropy on the reward gap with the label "the preferred one won":

L_RM(φ) = −E_{(x, yw, yl)} log σ(r_φ(x, y_w) − r_φ(x, y_l))

Only differences matter, so adding a constant to every reward changes nothing; InstructGPT normalizes the reward model so that the demonstrations score zero on average before reinforcement learning. InstructGPT also had labelers rank between K = 4 and K = 9 responses per prompt, which gives C(K, 2) comparisons from one prompt, and trained on all comparisons from a prompt together as one batch element, scaling the loss by 1/C(K, 2): treating them as independent samples made the reward model overfit, since each response appears in K − 1 of them.

RLHF: PPO with a KL penalty

With a reward model in hand, the policy is trained to maximize reward while staying close to the reference policy π_ref, the supervised model it started from:

max_π E_{x, y∼π}[ r_φ(x, y) ] − β · KL(π(·|x) ‖ π_ref(·|x))

In practice the KL term is folded into the reward. Each sampled response is scored once by r_φ at its end, and every token pays −β · log(π(y_t|·) / π_ref(y_t|·)); summed over the response this is a one-sample estimate of the sequence KL. The resulting per-token rewards go to PPO, with a value head estimating advantages and the clipped ratio keeping each update small. InstructGPT also added a pretraining term, "PPO-ptx", that mixes ordinary language-modeling gradients on pretraining data into the PPO update to reduce regressions on public benchmarks.

The KL term does two jobs. It keeps the policy in the region where the reward model was trained, since the reward model has only seen responses from something like π_ref, and it stops the policy collapsing onto a single high-reward response: without it, the optimum of E[r] is a point mass on the arg max.

DPO: the closed-form optimum, turned around

For a fixed prompt, the KL-regularized objective has an exact solution. Write the objective as −β times a KL to a tilted distribution plus a constant, and the maximizer is that distribution:

π*(y|x) = π_ref(y|x) · exp(r(x, y) / β) / Z(x),   Z(x) = Σ_y π_ref(y|x) · exp(r(x, y) / β)

Z(x) sums over every possible response, so this formula cannot be used to sample; that is why RLHF needs PPO. But it can be solved for the reward. Take logs:

r(x, y) = β · log(π*(y|x) / π_ref(y|x)) + β · log Z(x)

Substitute this into Bradley–Terry. The β log Z(x) term is the same for y_w and y_l, so it cancels in the difference, and the preference probability depends only on the policy and the reference:

P(y_w ≻ y_l | x) = σ(β · log(π*(y_w|x) / π_ref(y_w|x)) − β · log(π*(y_l|x) / π_ref(y_l|x)))

Now replace π* with the trainable π_θ and maximize the likelihood of the observed preferences. That is Direct Preference Optimization:

L_DPO(θ) = −E log σ(β · log(π_θ(y_w|x) / π_ref(y_w|x)) − β · log(π_θ(y_l|x) / π_ref(y_l|x)))

The quantity r̂_θ(x, y) = β · log(π_θ(y|x) / π_ref(y|x)) is the policy's implicit reward; the paper's title says it: your language model is secretly a reward model. The loss is the reward-model loss with r̂_θ in place of r_φ, so fitting it fits a reward model whose optimal policy is π_θ itself. Its gradient is

∇L_DPO = −β · E[ σ(r̂_θ(y_l) − r̂_θ(y_w)) · (∇ log π_θ(y_w|x) − ∇ log π_θ(y_l|x)) ]

It pushes up the preferred response and down the rejected one, weighted by how wrong the implicit reward currently is: pairs the model already orders correctly by a wide margin contribute almost nothing. The paper reports that dropping the weight and just pushing the pair apart degenerates the model. Every log π(y|x) is a sum of token log-probabilities, the same quantity pretraining maximizes, so DPO needs four forward passes per pair (policy and reference, on both responses) and no sampling.

Variants: IPO, KTO and GRPO

IPO (Azar et al.) starts from a weakness in DPO's derivation. If a pair is always won by the same response, Bradley–Terry says the reward gap is infinite, and DPO keeps widening the log-ratio margin h = log(π_θ(y_w)/π_ref(y_w)) − log(π_θ(y_l)/π_ref(y_l)) forever, so the KL regularization stops binding on finite data. IPO regresses the margin to a fixed target instead:

L_IPO(θ) = E (h_θ(x, y_w, y_l) − 1/(2τ))²

where τ plays the role of β. The worked example shows both behaviours on one unanimous pair.

KTO (Ethayarajh et al.) drops the pairs. It needs only a label per response, desirable or undesirable, which is easier to collect, for example from thumbs-up and thumbs-down. Its loss is built on the value function of Kahneman and Tversky's prospect theory: the implicit reward β · log(π_θ/π_ref) of each response is compared to a reference point estimated from the KL between policy and reference, and gains and losses are weighted separately, which lets the loss be tuned when desirable and undesirable examples are imbalanced.

GRPO (Group Relative Policy Optimization, from DeepSeekMath) keeps the reinforcement learning but removes PPO's value network. For each prompt it samples a group of G responses, scores each, and uses the group-normalized score as every token's advantage:

Â_i = (r_i − mean(r_1..G)) / std(r_1..G)

The clipped PPO ratio objective is unchanged. Instead of subtracting a KL penalty from the reward, GRPO adds a KL term directly to the loss, estimated per token with the unbiased, always non-negative estimator k3 = π_ref/π_θ − log(π_ref/π_θ) − 1. With a rule-based reward, such as checking a math answer, GRPO needs neither a value model nor a learned reward model.

Reward hacking

A reward model is a proxy fitted to a finite set of judgments, and the policy is an optimizer pointed at it. As optimization pushes the policy away from π_ref, it finds responses the proxy over-rates: longer answers, confident tone, flattering agreement, formatting the labelers happened to like. This is Goodhart's law. Gao, Schulman and Hilton measured it by training policies against a proxy reward model and scoring them with a larger "gold" reward model: as the KL from the initial policy grows, the proxy reward keeps rising while the gold reward rises, peaks and falls, and larger reward models peak later. The worked example below reproduces the shape in miniature.

The defenses are the ones already on this page plus some process: the KL coefficient, early stopping on a held-out judgment, larger and better reward models, fresh comparisons on the current policy's outputs, ensembles of reward models, and penalties for known biases such as length. DPO is not immune: its implicit reward is fitted to the same biased judgments, and its optimum is the same policy.

Worked example

One prompt, "Why is the sky blue?", and four candidate responses, small enough to solve exactly. The reference policy is a probability over whole responses, π_ref = (0.40, 0.25, 0.10, 0.25). Each pair was compared ten times, and the counts are invented to carry a known flaw: labelers somewhat prefer the longer answers, so the padded response C, which contains a factual error, wins more often than it should. A hidden "true" reward, (1.0, 1.5, −0.5, −1.5), stands in for the gold judge.

The Goodhart curve in miniature. Each point is the exact optimum for one β. Past β = 0.5 the policy keeps buying proxy reward by moving toward C, the padded answer the labelers over-rated, and by β = 0.05 it is worse than where it started.

The DPO loss is the reward-model loss on the implicit reward gap, so it has the same shape: a pair ordered the wrong way gets nearly full weight, a pair already ordered by a wide margin gets almost none. The loss only flattens toward zero as m grows, which is why a unanimous pair keeps pushing.

Implementation

import torch
import torch.nn.functional as F

def response_logp(model, input_ids, response_mask):
    # model returns logits (B, T, V); response_mask is 1 on response tokens, 0 on the prompt and padding
    logits = model(input_ids)[:, :-1].float()
    targets = input_ids[:, 1:].unsqueeze(-1)
    token_logp = F.log_softmax(logits, dim=-1).gather(-1, targets).squeeze(-1)
    return (token_logp * response_mask[:, 1:]).sum(-1)          # log pi(y|x), shape (B,)

def reward_model_loss(reward_model, chosen, rejected):
    gap = reward_model(**chosen) - reward_model(**rejected)       # one scalar per sequence
    return -F.logsigmoid(gap).mean()

def dpo_loss(policy, reference, chosen, rejected, beta=0.1):
    pi_w, pi_l = response_logp(policy, **chosen), response_logp(policy, **rejected)
    with torch.no_grad():
        ref_w, ref_l = response_logp(reference, **chosen), response_logp(reference, **rejected)
    reward_w, reward_l = beta * (pi_w - ref_w), beta * (pi_l - ref_l)   # implicit rewards
    loss = -F.logsigmoid(reward_w - reward_l).mean()
    accuracy = (reward_w > reward_l).float().mean()
    return loss, accuracy

def ipo_loss(pi_w, pi_l, ref_w, ref_l, tau=0.1):
    h = (pi_w - ref_w) - (pi_l - ref_l)
    return ((h - 1 / (2 * tau)) ** 2).mean()

def grpo_advantages(rewards, eps=1e-6):                         # rewards: (prompts, G)
    return (rewards - rewards.mean(-1, keepdim=True)) / (rewards.std(-1, keepdim=True) + eps)

def k3(logp, ref_logp):                                         # per token, sampled from the policy
    log_ratio = ref_logp - logp
    return torch.exp(log_ratio) - log_ratio - 1

The reference model is a frozen copy of the starting policy, so DPO holds two models in memory; its log-probabilities can be computed once over the dataset and stored, since they never change. Sum log-probabilities only over response tokens: including the prompt adds the same constant to both sides of each ratio and does nothing but cost precision, and averaging instead of summing changes the objective. Implementations differ on the standard deviation in GRPO (PyTorch's std divides by n − 1 by default; the worked example above uses the population value) and on how token losses are averaged across responses of different lengths. Track the implicit-reward accuracy and margins during DPO, and the KL and reward during PPO: a KL that climbs while held-out human or gold-model preference stalls is the reward-hacking signal.

Tradeoffs

• Online or offline. PPO and GRPO learn from the current policy's own samples, so they can correct mistakes the dataset never showed; DPO, IPO and KTO only reweight responses already in the data. Iterated DPO, with fresh pairs from the current policy each round, closes part of that gap.
• β sets the leash. Small β lets the policy move far, which the sweep above shows can buy proxy reward at the expense of real quality; large β changes little. It is the main hyperparameter in every method here.
• The data decides what "preferred" means. Labeler instructions, the people chosen and their biases toward length or confidence all end up in the reward. No loss function fixes a preference dataset that rewards the wrong thing.

Connections

This page continues fine-tuning, which produces the reference policy, and reinforcement learning, which builds policy gradients and PPO's clipped ratio. The reward model is logistic regression on a reward gap, trained with cross-entropy, and every log π(y|x) is the quantity pretraining maximizes for a decoder-only model. The KL penalty is the entropy and KL divergence page put to work.

References

• Bradley and Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 1952. doi:10.2307/2334029
• Christiano et al. Deep Reinforcement Learning from Human Preferences. NeurIPS 2017. arXiv:1706.03741
• Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv:1707.06347
• Ziegler et al. Fine-Tuning Language Models from Human Preferences. 2019. arXiv:1909.08593
• Stiennon et al. Learning to Summarize from Human Feedback. NeurIPS 2020. arXiv:2009.01325
• Ouyang et al. Training Language Models to Follow Instructions with Human Feedback. NeurIPS 2022. arXiv:2203.02155
• Gao, Schulman and Hilton. Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760
• Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290
• Azar et al. A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS 2024. arXiv:2310.12036
• Ethayarajh et al. KTO: Model Alignment as Prospect Theoretic Optimization. 2024. arXiv:2402.01306
• Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. 2024. arXiv:2402.03300
• PyTorch v2.15.0: F.logsigmoid, F.log_softmax, torch.gather, F.binary_cross_entropy_with_logits, F.kl_div, F.cross_entropy.