Autoregressive models

Problem

We want a model of a whole sequence x = (x₁, …, x_T): a sentence, an image read pixel by pixel, a second of audio. We want to assign it a probability (to score data and train by cross-entropy) and to draw new samples from it. A direct table of p(x) is hopeless. With a vocabulary of V tokens there are V^T possible sequences; for a 50,000-token vocabulary and just three tokens that is already 125,000,000,000,000 entries. We need a way to break the joint probability into pieces a neural network can output, without assuming the pieces are independent.

Intuition

Read a sentence left to right and guess each next word. If you can say how likely every possible next word is, given the words so far, you have described the whole sentence's probability: multiply your guesses' probabilities together. Generating is the same game played forward: draw a word, write it down, and guess again with that word now part of the past. The model only ever answers one question, "what comes next?", but it answers it with the whole past in view.

Training sees only true prefixes; generation sees the model's own. That gap is exposure bias.

Mechanics

The chain rule factorization

For any joint distribution, by the definition of conditional probability applied repeatedly,

p(x₁, …, x_T) = p(x₁) · p(x₂ | x₁) · … · p(x_T | x₁, …, x_T−1) = Π_t p(x_t | x_<t).

This is an identity, not an approximation: an autoregressive model that represented every conditional exactly would represent the joint exactly. The approximations come from the model: a Markov model keeps only the last k tokens of the past; an RNN squeezes the past into a fixed-size state; a Transformer with a causal mask reads the whole past (up to its context length) and shares one set of weights across all positions. Any order works in principle (right to left, or a scan through image pixels), but the model must fix one. MADE builds the same factorization into a single feed-forward autoencoder by masking its weights so each output only sees earlier inputs.

The log of the product is a sum, so the log-likelihood of a sequence is Σ_t log p(x_t | x_<t): one classification loss per position. Sampling is ancestral: draw x₁, then x₂ given it, and so on. Scoring a sequence takes one parallel pass; sampling takes T sequential passes.

Maximum likelihood by teacher forcing

Training maximises the log-likelihood of the data, which is the same as minimising the summed cross-entropy between the model's next-token distribution and the true next token. During training the model is fed the true prefix at every position, never its own guesses. That is teacher forcing. It makes training parallel: the input is the sequence, the target is the same sequence shifted one step left, and with a causal model every position's loss comes out of one forward pass. For a model that is just a table of conditional probabilities, the maximum-likelihood solution is the count ratio: p(b | a) = count(a b) / count(a ·). The worked example checks that the gradient of the loss is zero there.

Exposure bias

At generation time there is no teacher. The model conditions on what it wrote, and if it wrote something unusual it is now in a prefix it never saw in training. Errors can compound: one bad token makes the next prediction worse. This mismatch between training on true prefixes and running on generated ones is called exposure bias. Remedies include scheduled sampling (gradually feeding the model its own predictions during training), sequence-level training objectives, and in practice simply much more data and capacity, so that the model's own outputs stay close to the data distribution. Decoding rules (see decoding and sampling) also matter: truncating the long tail keeps the model away from prefixes it handles badly.

Causal convolutions: WaveNet

A convolution can be autoregressive if output t only reads inputs at t and earlier. PyTorch's nn.Conv1d pads both sides by padding, so it is not causal by itself. The usual trick is to pad only the left with F.pad(x, (dilation·(k−1), 0)) and use padding=0. Stacking causal layers grows the receptive field, the window of past inputs an output can see, by (k−1)·d per layer for kernel size k and dilation d. With dilations 1, 2, 4, … the receptive field doubles per layer instead of growing by one. WaveNet used kernel size 2 with dilations 1, 2, …, 512 and repeated that block, so one block sees 1024 samples.

Kernel [0.5, 0.5] at every layer; each node shows its computed value. Missing left neighbours read the zero padding.

Masked convolutions: PixelCNN

For images, fix a raster order (rows top to bottom, left to right in a row) and predict each pixel from the ones before it. PixelCNN keeps a normal k×k convolution but multiplies its kernel by a mask that zeroes every tap at or after the centre in raster order. The first layer uses mask A, which also zeroes the centre, so a pixel never sees itself. Later layers use mask B, which keeps the centre, because the centre position there holds features computed only from earlier pixels. (With colour, the paper also orders R, G, B within a pixel.) A 5×5 mask A has 12 live taps and mask B has 13.

Stacked masked 3×3 layers leave a blind spot: some earlier pixels up and to the right never reach the output, however many layers you stack, because each layer's receptive field only grows up-left and sideways in a way that misses a wedge. Gated PixelCNN fixed this by splitting the network into a vertical stack (rows above) and a horizontal stack (the current row to the left).

Language models: the main case

A modern decoder-only language model is an autoregressive model over tokens: a causal Transformer outputs logits for p(x_t | x_<t) at every position, trained with teacher forcing and cross-entropy on shifted targets, and sampled one token at a time. The KV cache exists because generation is sequential: it saves the past's keys and values so each new step does not recompute them.

Worked example

Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

class CausalConv1d(nn.Conv1d):
    def __init__(self, c_in, c_out, kernel_size, dilation=1):
        super().__init__(c_in, c_out, kernel_size, dilation=dilation, padding=0)
        self.left = dilation * (kernel_size - 1)

    def forward(self, x):                     # x: [B, C, T]
        return super().forward(F.pad(x, (self.left, 0)))   # output [B, C_out, T]

class MaskedConv2d(nn.Conv2d):                # PixelCNN, single channel ordering
    def __init__(self, mask_type, *args, **kwargs):
        super().__init__(*args, **kwargs)
        k = self.kernel_size[0]
        mask = torch.ones_like(self.weight)
        mask[:, :, k // 2, k // 2 + (mask_type == "B"):] = 0
        mask[:, :, k // 2 + 1:] = 0
        self.register_buffer("mask", mask)

    def forward(self, x):
        return F.conv2d(x, self.weight * self.mask, self.bias,
                        self.stride, self.padding, self.dilation, self.groups)

# teacher forcing for a language model: inputs and shifted targets
def lm_loss(model, ids):                      # ids: [B, T+1]
    logits = model(ids[:, :-1])               # [B, T, V], causal model
    return F.cross_entropy(logits.reshape(-1, logits.size(-1)),
                           ids[:, 1:].reshape(-1), ignore_index=-100)

@torch.no_grad()
def generate(model, ids, steps):
    for _ in range(steps):
        probs = model(ids)[:, -1].softmax(-1) # only the last position
        nxt = torch.multinomial(probs, 1)     # [B, 1]
        ids = torch.cat([ids, nxt], dim=1)    # fed back: no teacher now
    return ids

The causal convolution relies on Conv1d's default padding_mode="zeros" being irrelevant once padding=0; all padding comes from F.pad, whose tuple starts with the last dimension's (left, right). The generate loop recomputes the whole prefix each step for clarity; real implementations keep a cache.

Tradeoffs

• Exact likelihood. Autoregressive models give an exact log p(x), so they can be trained and compared by cross-entropy or perplexity directly. Many other generative models (diffusion models, VAEs) only give a bound or none.
• Sequential sampling. Scoring is parallel, but generating T tokens takes T steps. That is the cost behind the KV cache and speculative decoding, and why parallel generators exist.
• A fixed order. The model must commit to an order. Text has a natural one; images and audio do not, and the order affects what is easy to learn (PixelCNN's blind spot is an artefact of order plus architecture).
• Exposure bias. Teacher forcing is efficient and is maximum likelihood, but it never shows the model its own mistakes. Scheduled sampling breaks the parallelism of training; most large models live with the bias.
• Context length. Convolutions see a fixed receptive field; RNNs see everything through a bottleneck; Transformers see the whole window at quadratic attention cost.

Check yourself

Connections

References

• Bengio, Ducharme, Vincent and Jauvin. A Neural Probabilistic Language Model. JMLR 3, 2003 (JMLR).
• van den Oord, Kalchbrenner and Kavukcuoglu. Pixel Recurrent Neural Networks (PixelCNN, masks A and B). ICML, 2016 (arXiv 1601.06759).
• van den Oord, Kalchbrenner, Vinyals, Espeholt, Graves and Kavukcuoglu. Conditional Image Generation with PixelCNN Decoders (Gated PixelCNN, the blind spot). NeurIPS, 2016 (arXiv 1606.05328).
• van den Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior and Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. 2016 (arXiv 1609.03499).
• Bengio, Vinyals, Jaitly and Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NeurIPS, 2015 (arXiv 1506.03099).
• Ranzato, Chopra, Auli and Zaremba. Sequence Level Training with Recurrent Neural Networks (names exposure bias). ICLR, 2016 (arXiv 1511.06732).
• Germain, Gregor, Murray and Larochelle. MADE: Masked Autoencoder for Distribution Estimation. ICML, 2015 (arXiv 1502.03509).
• PyTorch v2.15.0: categorical.py, CrossEntropyLoss, Conv1d/Conv2d, F.pad.