Encoder-only models

Problem

Many language tasks are about understanding a piece of text that already exists: is this review positive, which words in this sentence are names, does this passage answer that question, are these two sentences paraphrases. For all of them the whole input is available up front, so there is no reason to hide the end of the sentence from its beginning. The word "bank" in "she sat on the bank of the river" is only resolved by words that come after it.

A decoder-only model can't use that right-hand context: its causal mask exists so it can generate. We want the opposite trade, a model whose every output vector is built from the entire input, plus a way to pre-train it on unlabelled text, because labelled data for each task is scarce. Next-token prediction doesn't work as that pre-training task once every token can see every other token: the answer would be sitting in the input.

Intuition

Think of a cloze test: a paragraph with some words blanked out, and you fill each blank using everything around it. An encoder-only model is trained on exactly that. Hide some tokens, let every position attend to every other position, and ask the model to recover the hidden ones. To do it well it has to build, at every position, a vector that summarises the word in its full context, and those vectors are what you keep and fine-tune for downstream tasks.

The architecture is the encoder half of the Transformer, unchanged: no causal mask, no cross-attention. Only two things are new, both about the input: a learned segment embedding so that one sequence can hold two sentences, and special tokens: [CLS] at the front, whose final vector stands for the whole input, and [SEP] after each sentence.

Segment A covers [CLS], the first sentence and its [SEP]; segment B covers the second sentence and the final [SEP]. A single-sentence input uses segment A throughout.

Mechanics

Bidirectional attention: delete the mask

Self-attention is the same computation as on the attention page; the only difference from a decoder is that no −∞ is added above the diagonal. Here are the attention weights for the four-token example (the, cat, sat, down) both ways:

Rows sum to 1 in both grids. Only the last token's row is identical, because it is the one position that already sees everything under the causal mask.

The price is the property decoders rely on. Drop the last token and rerun: with the causal mask the first three outputs don't move at all (gap 0.0); without it they change by up to 0.36. Every encoder output depends on the whole input, so you can't append a token and reuse earlier work, and you can't generate left to right. That is the trade: richer per-token vectors, no generation.

Padding and the key padding mask

A batch holds sequences of different lengths, so shorter ones are filled with [PAD] to a common width. In our batch the pair has 12 tokens and the single sentence [CLS] the cat sat [SEP] has 5, followed by 7 pads. Pads must be invisible: no real token should attend to them. PyTorch takes a src_key_padding_mask of shape [B, T] in which True means "ignore this key" (documented on MultiheadAttention); internally a boolean mask becomes a float mask with −∞ at the ignored keys (_canonical_mask).

The check: take the first three tokens of the attention example, append a pad whose key and value are zero, and attend. With the pad masked, the three real outputs equal the outputs of the three-token sequence alone, gap 0.0. Without the mask the pad still takes attention weight 0.165, 0.089, 0.043 from the three real rows (a zero key scores 0, which is not −∞) and the outputs move by up to 0.13. Forgetting the padding mask doesn't crash; it quietly makes a sentence's vectors depend on how long its batch-mates were.

The pad's own row still gets computed (here it averages the three real tokens), so it produces some vector; nothing downstream should read it. When TransformerEncoder runs in inference mode with a padding mask it can take a fast path that packs the batch into a nested tensor, skipping pads entirely, and then unpacks with zeros at the padded positions (lines 531-550), so pad outputs differ between training and evaluation. Another reason to never use them.

Masked language modelling

BERT's pre-training picks 15% of the token positions (implementations skip [CLS], [SEP] and pads) and asks the model to predict the original token at each. Of the chosen positions, 80% are replaced by [MASK], 10% by a random token and 10% are left as they are. The mix exists because [MASK] never appears at fine-tuning time: if it were the only corruption, the model could learn that only [MASK] positions need a good representation. Random and unchanged tokens force it to keep every position's vector honest.

The loss is cross-entropy over the vocabulary, but only at chosen positions. Every other label is set to −100, the default ignore_index of CrossEntropyLoss (line 1389); ignored targets contribute nothing, and the mean is taken over the rest (lines 1278-1281).

BERT also had a second objective, next sentence prediction: the [CLS] vector classifies whether sentence B really followed sentence A (half the pairs) or was drawn at random. It is why the input has segments. Later work such as RoBERTa reported that dropping it didn't hurt, and most successors pre-train with masked tokens alone.

The block, post-norm

BERT keeps the original Transformer's order: x ← LayerNorm(x + Attention(x)), then x ← LayerNorm(x + FFN(x)). That is PyTorch's default, norm_first=False (lines 745-755 list the defaults; lines 945-957 are the two branches). Because every block ends in a LayerNorm, no final norm is needed, unlike the pre-norm decoder. The feed-forward width is 4 × d with GELU, which PyTorch's layer accepts as activation="gelu" (lines 1204-1210); the layer's default is ReLU.

Using the outputs

The encoder returns a [B, T, d] tensor. Fine-tuning adds one small layer on top, chosen by the task: a classifier on the [CLS] vector for sentence or sentence-pair labels; a classifier on every token's vector for tagging (named entities, parts of speech); two vectors that score each token as the start or end of an answer span for extractive question answering. The whole model is then trained end to end for a few epochs at a small learning rate. For embeddings and search, a pooled vector (the [CLS] vector, or the mean of the non-pad token vectors) represents the whole text.

Worked example

Implementation

import torch
from torch import nn
import torch.nn.functional as F

class Encoder(nn.Module):
    def __init__(self, vocab=30000, positions=512, d=768, layers=12, heads=12, pad_id=0):
        super().__init__()
        self.tok = nn.Embedding(vocab, d, padding_idx=pad_id)
        self.pos = nn.Embedding(positions, d)
        self.seg = nn.Embedding(2, d)
        self.norm = nn.LayerNorm(d)
        self.drop = nn.Dropout(0.1)
        block = nn.TransformerEncoderLayer(
            d, heads, dim_feedforward=4 * d, dropout=0.1,
            activation="gelu", batch_first=True,          # norm_first=False: post-norm
        )
        self.blocks = nn.TransformerEncoder(block, layers)
        self.mlm = nn.Linear(d, vocab)
        self.pad_id = pad_id

    def forward(self, ids, segments):                    # both [B, T]
        T = ids.size(1)
        pos = torch.arange(T, device=ids.device)
        x = self.drop(self.norm(self.tok(ids) + self.pos(pos) + self.seg(segments)))
        pad = ids == self.pad_id                         # True = ignore this key
        return self.blocks(x, src_key_padding_mask=pad)  # [B, T, d]

def mask_tokens(ids, special, vocab, mask_id, rate=0.15):
    labels = ids.clone()
    chosen = (torch.rand(ids.shape) < rate) & ~special
    labels[~chosen] = -100                               # ignored by cross_entropy
    roll = torch.rand(ids.shape)
    inputs = ids.clone()
    inputs[chosen & (roll < 0.8)] = mask_id
    swap = chosen & (roll >= 0.8) & (roll < 0.9)
    inputs[swap] = torch.randint(vocab, (int(swap.sum()),))
    return inputs, labels                                # the last 10% stay as they are

model = Encoder()
ids = torch.randint(5, 30000, (2, 12))
special = torch.zeros_like(ids, dtype=torch.bool)
special[:, 0] = True                                     # the [CLS] slot
inputs, labels = mask_tokens(ids, special, 30000, mask_id=4)
hidden = model(inputs, torch.zeros_like(ids))
loss = F.cross_entropy(model.mlm(hidden).reshape(-1, 30000), labels.reshape(-1))
cls_vector = hidden[:, 0]                                # [B, d], for a classifier head

Things to know. padding_idx zeroes the pad row at construction (lines 179-186) and stops it from updating. The encoder takes the padding mask as src_key_padding_mask, converted with _canonical_mask (lines 433-439); a sequence's real tokens should come first, which the fast path checks when mask_check=True. Selection here is a coin flip per token, so the count varies around 15%; the page's helper instead samples an exact count. A real pipeline also never draws [CLS], [SEP] or pads, which is what special is for. Fine-tuning replaces self.mlm with a head for the task, often reading hidden[:, 0].

Tradeoffs

• Encoder-only versus decoder-only. For the same size, a bidirectional encoder usually gives better vectors for classification, tagging and retrieval, and a whole input is processed in one parallel pass. It can't generate, and it can't reuse work when the input grows by one token. Decoders trade some understanding per parameter for one model that does every task as text, which is why they took over once models became large.
• Sample efficiency. Only about 15% of positions carry a loss, against nearly 100% for next-token prediction. Variants such as ELECTRA score every position (was this token replaced?) to recover the signal.
• The [MASK] mismatch. Pre-training inputs contain a token fine-tuning inputs never do. The 80/10/10 split softens it; it doesn't remove it.
• Length. A learned table of 512 positions is a hard cap, and full attention costs grow with T². Long documents get split into windows.
• Pitfalls. A missing padding mask (outputs depend on batch-mates), reading pad outputs, labelling unmasked positions instead of −100 (the model learns to copy), and using the raw [CLS] vector of a model that was never fine-tuned as a sentence embedding, which often works poorly without training for that purpose.

Check yourself

Connections

References

• Devlin, Chang, Lee and Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019 (arXiv 1810.04805).
• Liu et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019 (arXiv 1907.11692).
• Clark, Luong, Le and Manning. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. ICLR, 2020 (arXiv 2003.10555).
• Vaswani et al. Attention Is All You Need. NeurIPS, 2017 (arXiv 1706.03762).
• Hendrycks and Gimpel. Gaussian Error Linear Units (GELUs). 2016 (arXiv 1606.08415).
• PyTorch v2.15.0: transformer.py, activation.py, sparse.py, loss.py, functional.py.