Multimodal models

Problem

A vision transformer trained on ImageNet knows a thousand labels, and a language model knows text but has never seen a pixel. Many tasks need both: search photos by a sentence, describe an image, answer a question about a chart, transcribe and then reason about a recording. Collecting labels for each of these is expensive, and a fixed label set cannot cover "a corgi wearing sunglasses on a skateboard".

The web, however, is full of images next to text that describes them: alt text, captions, titles. Multimodal models turn that weak, noisy pairing into supervision. The two problems are how to learn a shared space in which an image and its caption land close together, and how to feed what an image encoder sees into a language model so the language model can talk about it.

Intuition

For the first, use a two-tower model: one encoder per modality, each producing a vector, trained so that matching pairs have a high dot product and mismatched pairs a low one. Every other caption in the batch is a negative for each image. The model never has to generate text; it only has to tell which caption goes with which image, a task that already forces it to recognize objects, scenes, styles and text in the image. Once trained, any label can be turned into a caption ("a photo of a dog") and classified by similarity, with no training examples at all.

For the second, treat the image encoder's output vectors as if they were words. Either project them into the language model's embedding space and put them in the sequence in front of the text, so ordinary self-attention reads them, or keep them separate and add cross-attention layers in which the text queries the image. The first is simple and is what most open models do; the second keeps the text sequence short and the language model untouched.

Mechanics

Contrastive image–text pretraining

CLIP (Radford et al., 2021) has an image encoder f, a ResNet or a ViT, and a text encoder g, a Transformer whose output at the end-of-text token summarizes the caption. Each is followed by a linear projection into a shared space of the same width, and both outputs are L2-normalized, so their dot product is a cosine similarity. For a batch of N pairs with unit vectors u_i = f(image_i)/‖·‖ and v_j = g(text_j)/‖·‖, the logits are

with t a learned scalar: s is an inverse temperature, initialized to 1/0.07 ≈ 14.29 and clipped at 100. Each row of S is a classification over N captions with the correct one on the diagonal, and each column a classification over N images. The loss is the average of the two cross-entropies:

This is InfoNCE (van den Oord et al., 2018) in both directions. It is a lower bound on the mutual information between the two views that tightens as N grows, which is one reason CLIP trained with 32,768 pairs per batch. Its gradient with respect to each logit follows from the softmax–cross-entropy derivative, p − y, applied to each direction:

where P^row is the softmax of S along each row and P^col along each column. Every diagonal entry is pulled up and every off-diagonal entry pushed down in proportion to how much probability it steals, so the gradient is spent on the hardest negatives. A larger s makes the softmax sharper: it concentrates on those hard negatives, and lets the loss approach zero once the diagonal wins; too small an s caps how confident the model can be at all.

Without labels, each row needs the others as negatives, so the loss depends on the whole batch. In data-parallel training each device gathers the embeddings from every other device to build the full N × N matrix. SigLIP (Zhai et al., 2023) replaces the softmax with an independent sigmoid on every pair, a binary "match or not" per entry, which removes the normalization over the batch and so the need to see all of it at once.

Zero-shot classification

To classify into K labels, embed one caption per label, such as "a photo of a {label}.", and pick the label whose text vector has the highest cosine with the image. The K text vectors act as the weights of a linear classifier that was never trained on the task. Prompt wording matters, since web captions rarely consist of a bare label; CLIP's authors averaged the normalized embeddings of many prompts per class ("a blurry photo of a …", "a drawing of a …") into one vector, a prompt ensemble, which is computed once and costs nothing at inference.

Vision-language models

A vision-language model (VLM) generates text conditioned on images. The usual recipe has three parts: a pretrained image encoder, often a CLIP or SigLIP ViT; a connector that maps its outputs into the language model's embedding space; and a pretrained decoder-only language model. LLaVA (Liu et al., 2023) used a single linear layer as the connector, and LLaVA-1.5 a two-layer MLP, on the patch tokens of a CLIP ViT-L/14 at 336 pixels: (336/14)² = 576 tokens per image. Training typically runs in two stages: first only the connector, on image–caption pairs, to align the spaces; then the connector and the language model together on instruction data (see fine-tuning).

Higher resolution means more tokens. Splitting a large image into tiles at the encoder's native resolution, plus one downscaled overview, multiplies the count: four tiles and an overview give 2,880 tokens. Connectors shrink it back by pooling neighbouring patches (2 × 2 pooling gives 144) or by a small set of learned queries that cross-attend to all patches and return a fixed number of tokens, like Flamingo's Perceiver Resampler with 64.

Concatenation versus cross-attention

Token concatenation (LLaVA and most open VLMs) places the projected image tokens in the input sequence, so every self-attention layer of the language model attends over them as over any text. Nothing in the language model changes. The cost is sequence length: every layer's attention and every layer's KV cache includes the image tokens.

Gated cross-attention (Flamingo, Alayrac et al., 2022) freezes the language model and inserts new blocks between its layers. In each, the text hidden states are queries and the visual tokens are keys and values, so attention is rectangular, text length by image length, and the text sequence never grows. The block's output is added through a gate:

with α a learned scalar initialized to 0. At the start, tanh(0) = 0 and the model is exactly the frozen language model, so training begins from its full ability and opens the gate gradually. Flamingo also masks the cross-attention so each text token sees only the image that immediately precedes it, which lets it read interleaved sequences of many images and text. Llama 3's vision models use the same adapter idea of cross-attention layers added to a language model.

Audio tokens

Audio enters the same way, as a sequence of vectors. Whisper (Radford et al., 2022) resamples speech to 16,000 Hz, computes an 80-channel log-Mel spectrogram with a 10 ms hop over 30-second windows, and runs two convolutions, the second with stride 2, before a Transformer encoder; a decoder cross-attends to it to emit text. Audio-language models attach such an encoder to a language model with a projector, exactly like a VLM.

To generate audio, a model needs discrete output tokens. Neural codecs such as SoundStream and EnCodec (Défossez et al., 2022) encode the waveform into frames and quantize each frame with residual vector quantization: a first codebook picks the nearest code, a second quantizes what is left, and so on. More codebooks give higher fidelity and more tokens per second. A language model trained on those tokens (AudioLM, VALL-E, and speech-native assistants) can continue speech the way a text model continues a sentence, usually predicting the first codebook first and the finer ones from it.

Worked example

Implementation

The CLIP loss in PyTorch. The encoders are any image and text models with pooled outputs; everything that is specific to CLIP is these lines.

import math
import torch
import torch.nn.functional as F
from torch import nn

class ClipHead(nn.Module):
    def __init__(self, image_dim, text_dim, embed_dim=512):
        super().__init__()
        self.image_proj = nn.Linear(image_dim, embed_dim, bias=False)
        self.text_proj = nn.Linear(text_dim, embed_dim, bias=False)
        self.logit_scale = nn.Parameter(torch.tensor(math.log(1 / 0.07)))

    def forward(self, image_features, text_features):     # [N, image_dim], [N, text_dim]
        u = F.normalize(self.image_proj(image_features), dim=-1)
        v = F.normalize(self.text_proj(text_features), dim=-1)
        logits = self.logit_scale.exp().clamp(max=100) * u @ v.T    # [N, N]
        target = torch.arange(len(logits), device=logits.device)
        return (F.cross_entropy(logits, target) + F.cross_entropy(logits.T, target)) / 2

class GatedCrossAttention(nn.Module):
    def __init__(self, d_model, heads):
        super().__init__()
        self.heads = heads
        self.norm = nn.LayerNorm(d_model)
        self.q = nn.Linear(d_model, d_model, bias=False)
        self.kv = nn.Linear(d_model, 2 * d_model, bias=False)
        self.out = nn.Linear(d_model, d_model, bias=False)
        self.alpha = nn.Parameter(torch.zeros(()))

    def forward(self, x, image):                          # [B, L, d], [B, S, d]
        B, L, d = x.shape
        q = self.q(self.norm(x)).view(B, L, self.heads, -1).transpose(1, 2)
        k, v = self.kv(image).view(B, image.shape[1], 2, self.heads, -1).permute(2, 0, 3, 1, 4)
        y = F.scaled_dot_product_attention(q, k, v)        # [B, H, L, d/H]: L queries over S keys
        return x + self.alpha.tanh() * self.out(y.transpose(1, 2).reshape(B, L, d))

The clamp sits on the scale, not on the parameter, so the parameter keeps receiving gradients but the logits cannot exceed 100 times a cosine. In data-parallel training, gather u and v from all ranks (with a gather that passes gradients back) before forming the logits, or each device sees only its local negatives. For concatenation, the connector is just nn.Sequential(nn.Linear(d_vision, d_model), nn.GELU(), nn.Linear(d_model, d_model)) and its output is spliced into inputs_embeds where the image placeholder tokens were. A full Flamingo block also has a gated feed-forward layer after the cross-attention.

Tradeoffs

• Contrastive vs generative pretraining. A contrastive model gives embeddings for retrieval and zero-shot classification and is cheap to run per item, but cannot describe an image. Captioning objectives learn more fine-grained detail and generate text, at a higher cost per example. Many models combine both.
• Softmax vs sigmoid loss. The softmax over the batch needs large batches and an all-gather; SigLIP's pairwise sigmoid works with smaller batches and less communication but treats each pair independently.
• Batch size. More in-batch negatives tighten the InfoNCE bound and give harder negatives, but the logit matrix grows as N² and the gathered embeddings with N.
• Concatenation vs cross-attention. Concatenation needs no change to the language model and lets every layer attend to image detail, but image tokens fill the context window and the KV cache. Cross-attention keeps the sequence short and the language model frozen, at the price of new parameters and a less direct interaction.
• Resolution vs tokens. Documents, charts and small text need high resolution, which multiplies image tokens; pooling and resamplers save compute but lose detail. Tiling with a low-resolution overview is the usual compromise.
• Frozen vs trained encoder. Freezing the image encoder protects its general features and saves memory; unfreezing it late in training improves fine detail at the risk of forgetting.
• Web data. Alt text is noisy, biased and sometimes harmful; filtering and deduplication matter as much as model size, and zero-shot classifiers inherit the associations in their captions.

Connections

References

• Radford, Kim, Hallacy et al. Learning Transferable Visual Models From Natural Language Supervision. ICML 2021. arXiv:2103.00020
• Alayrac, Donahue, Luc et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. arXiv:2204.14198
• van den Oord, Li and Vinyals. Representation Learning with Contrastive Predictive Coding. 2018. arXiv:1807.03748
• Zhai, Mustafa, Kolesnikov and Beyer. Sigmoid Loss for Language Image Pre-Training. ICCV 2023. arXiv:2303.15343
• Liu, Li, Wu and Lee. Visual Instruction Tuning. NeurIPS 2023. arXiv:2304.08485; and Improved Baselines with Visual Instruction Tuning, arXiv:2310.03744
• Radford, Kim, Xu et al. Robust Speech Recognition via Large-Scale Weak Supervision. (Whisper) 2022. arXiv:2212.04356
• Défossez, Copet, Synnaeve and Adi. High Fidelity Neural Audio Compression. (EnCodec) 2022. arXiv:2210.13438
• PyTorch v2.15.0: F.normalize, F.cross_entropy, F.scaled_dot_product_attention.