Vectors and dot products

Problem

Models represent words, images, users and products as vectors of numbers called embeddings. Once everything is a vector, the questions a model asks become geometric: is this document about the same thing as this query? Which earlier word should this word pay attention to? How much of this input points in the direction a neuron is looking for?

We need a cheap, differentiable way to compare two vectors, one that a GPU can do billions of times a second. The dot product is that tool, and knowing exactly what it measures (and what it doesn't) explains choices like scaling attention scores or normalizing embeddings before search.

Intuition

Read a vector [x, y] as an arrow from the origin to the point (x, y). It has a direction and a length. The dot product of two arrows is large and positive when they point the same way, zero when they are at right angles, and negative when they point apart. It also grows with the lengths of both arrows, which is the detail that matters most in practice.

d1 and q point the same way, so their cosine is 1. d4 is far off-direction but long, so its dot product (6) beats d2's (4) even though d2 is the closer match in direction.

Mechanics

Vectors as points and directions

A vector v in ℝⁿ is n numbers. In PyTorch it is a rank-1 tensor of shape [n], and a batch of them is [batch, n] (see Tensors, shapes and broadcasting). Adding vectors adds arrows tip to tail; multiplying by a scalar stretches an arrow without turning it. Embedding sizes are 64 to several thousand, but every idea on this page is the same in 2-D.

Norms: the length of a vector

‖v‖_p = (Σ_i |v_i|^p)^1/p

The L2 norm (p = 2) is ordinary length: ‖[3, 4]‖₂ = 5. The L1 norm sums absolute values (‖[3, −4]‖₁ = 7), and the L∞ norm takes the largest (‖[3, −4]‖_∞ = 4). L2 shows up in distances and weight decay, L1 in sparsity penalties, L∞ in gradient clipping by value and adversarial bounds (torch.linalg.vector_norm). Dividing a vector by its L2 norm gives a unit vector: same direction, length 1.

The dot product

u · v = Σ_i u_i v_i = ‖u‖ ‖v‖ cos θ

The first form is how it is computed: multiply matching entries and add. The second is what it means: lengths times the cosine of the angle θ between them. The two agree in any number of dimensions, which is why "similar direction" makes sense for 4096-wide embeddings. torch.dot takes exactly two 1-D tensors of the same length and, unlike NumPy's dot, nothing else (torch.dot); its docstring's own example is [2, 3] · [2, 1] = 7. For complex vectors torch.vdot conjugates the first argument (vdot), and torch.inner generalizes to the last axis of any two tensors (inner).

Projection

The dot product with a unit vector is the length of the shadow one vector casts on the other's line. Dividing by ‖a‖² instead of ‖a‖ gives the shadow as a vector:

proj_a(b) = (a · b / a · a) a

What's left over, b − proj_a(b), is orthogonal to a. That split is how Gram–Schmidt builds orthogonal bases and how least squares finds the closest point on a line, which the Linear regression page uses.

Cosine similarity and orthogonality

Divide the dot product by both lengths and only the angle is left:

cos(u, v) = u · v / (‖u‖ ‖v‖) ∈ [−1, 1]

1 means same direction, 0 means orthogonal (at right angles, sharing no direction), −1 means opposite. In high dimensions random vectors are nearly orthogonal, so a cosine of 0.3 between two embeddings can already be a strong signal. F.cosine_similarity clamps each norm separately to at least eps = 1e-8 before dividing, so a zero vector gives 0 instead of NaN (F.cosine_similarity); the module wrapper only stores dim and eps and calls it (nn.CosineSimilarity). Note that the module's docstring writes the clamp around the product of the norms, while the function it calls clamps each norm; the function is what runs.

Why attention and retrieval are dot products

• A neuron is a dot product. A linear layer's output j is w_j · x + b_j: how much the input points along the weight row the neuron has learned. A whole layer is many dot products at once, which is matrix multiplication, the next page.
• Attention scores are dot products. Each query is dotted with every key, the scores are scaled by 1/√d_k and softmaxed. The scaling exists precisely because dot products grow with dimension (see The Transformer).
• Retrieval is a dot product. Encode the query and every document, then return the documents with the highest q · d. Normalize everything first and that is cosine similarity; vector indexes exist to find the top scores without computing all of them.

Worked example

Implementation

import torch
import torch.nn.functional as F

q = torch.tensor([1., 2.])
docs = torch.tensor([[2., 4.], [2., 1.], [-2., 1.], [6., 0.]])

torch.dot(q, docs[1])                       # tensor(4.)
docs @ q                                    # tensor([10.,  4.,  0.,  6.]): all four at once
torch.linalg.vector_norm(docs, dim=1)       # lengths of each row
F.cosine_similarity(q.unsqueeze(0), docs)   # q broadcasts against every row
torch.cdist(q.unsqueeze(0), docs)           # [1, 4] distances

unit = F.normalize(docs, dim=1)             # divide each row by its L2 norm
(unit @ F.normalize(q, dim=0)).topk(2)      # cosine top-2: the core of vector search

a, b = torch.tensor([3., 1.]), torch.tensor([2., 2.])
along = (a @ b) / (a @ a) * a               # tensor([2.4000, 0.8000])
(b - along) @ a                             # tensor(0.): the residual is orthogonal

From scratch, in plain Python:

def dot(u, v):
    return sum(a * b for a, b in zip(u, v))

def norm(v):
    return dot(v, v) ** 0.5

def cosine(u, v, eps=1e-8):
    return dot(u, v) / (max(norm(u), eps) * max(norm(v), eps))

Tradeoffs

• Dot product versus cosine. The raw dot product lets length carry meaning: a recommender may want popular items, whose embeddings grow long, to score higher. Cosine removes length, which is fairer across documents of different sizes but throws that signal away. Many retrieval models train with normalized embeddings so the two coincide.
• Cosine versus Euclidean distance. For unit vectors they rank identically. For raw vectors, distance treats "near the same point" as similar, which suits clustering in a space where magnitude means something.
• Scale grows with dimension. The dot product of two random vectors with unit-variance entries has variance n. Attention divides by √d_k to undo this; forget it and softmax saturates.
• Precision. In float16, a long dot product can overflow or lose small terms. PyTorch kernels accumulate in higher precision, but hand-written loops in low precision may not.
• The eps clamp. It prevents NaN for zero vectors but means a tiny, nearly-zero vector gets a meaningless cosine. Filter degenerate embeddings before search.

Check yourself

Connections

References

• Strang. Introduction to Linear Algebra, 6th ed., chapter 1 (vectors, lengths and dot products). Wellesley-Cambridge Press, 2023.
• Vaswani et al. Attention Is All You Need. 2017, §3.2.1 on scaled dot-product attention. arXiv:1706.03762
• Karpukhin et al. Dense Passage Retrieval for Open-Domain Question Answering. 2020. arXiv:2004.04906. Retrieval as a dot product between query and passage encoders.
• PyTorch v2.15.0: F.cosine_similarity and torch/nn/modules/distance.py.