How a GPU runs a model

Problem

A GPU is advertised by its peak FLOP/s, yet a model rarely gets close. One layer runs at near peak, the next at a fraction of a percent; adding a cheap-looking activation costs more time than it should; a timer around a forward pass reports a number that is far too small. All of these come from two facts about the hardware: work is queued and runs asynchronously from the Python that queued it, and most operations are limited not by arithmetic but by how fast data moves to and from memory. This page builds a working model of both, enough to predict which operations are slow before profiling them.

Intuition

Think of the GPU as a very wide factory with a small loading dock. The factory floor (thousands of arithmetic units) can do enormous amounts of work per second, but every item has to come in and leave through the dock (memory bandwidth). A job that does a lot of work on each item that comes in, like a matrix multiplication, keeps the floor busy. A job that does one tiny thing to each item, like adding a constant, spends almost all its time waiting at the dock.

Meanwhile the factory takes orders from one clerk, the Python thread. The clerk writes each order (a kernel launch) and goes straight back to writing the next one without waiting for the first to finish. If each order is big, the floor is always busy and the clerk runs far ahead. If each order is tiny, the floor finishes it before the clerk has written the next one, and the expensive factory sits idle waiting on paperwork.

Amber blocks are the CPU's launch work (an assumed 5 µs per kernel), green blocks the kernels themselves. With short kernels the GPU waits for each launch; with long ones the CPU races ahead, and a timer read on the CPU at the dashed line would report 50 µs for 505 µs of work.

Mechanics

Kernels

A kernel is one function the GPU runs over many threads at once. Calling x + y on CUDA tensors launches one elementwise-add kernel; a @ b launches a matrix-multiplication kernel picked from a vendor library; a fused attention call is one large kernel doing what would otherwise take several. The launch describes a grid of thread blocks; the GPU spreads the blocks over its streaming multiprocessors (SMs), each of which runs many threads in lockstep groups. A launch has a fixed cost on the CPU side (Python, PyTorch's dispatcher, the driver) that does not shrink with the size of the work, which is why a long chain of tiny operations is slow no matter how fast the GPU is.

Streams and asynchronous execution

A stream is a queue of GPU work. In PyTorch's words it is "a linear sequence of execution that belongs to a specific device, independent from other streams": kernels on one stream run in the order they were queued, and kernels on different streams may overlap. Each device has a default stream, and unless code selects another one with torch.cuda.stream(s), every operation goes there, so ordinary PyTorch code is correctly ordered without thinking about streams at all.

Queuing is asynchronous: a CUDA operation returns to Python as soon as its kernel is enqueued, and the result tensor is a promise that the data will be there by the time a later kernel on the same stream reads it. The CPU only waits at a synchronization point, which is anything that needs a value on the host:

• tensor.item(), tensor.tolist(), print(tensor) and copies to the CPU wait for the kernels that produce the value.
• torch.cuda.synchronize() waits "for all kernels in all streams on a CUDA device to complete"; Stream.synchronize() and Event.synchronize() wait for one stream or for the work an event captured.
• Data-dependent shapes (torch.nonzero, boolean-mask indexing) need the result size on the CPU before they can allocate it.

Every unneeded synchronization point stalls the queue: the GPU drains, the CPU wakes, and the next launch starts from an empty pipeline. Logging loss.item() every step is the classic example. Work across streams is ordered with events: event.record(s1) marks a point in stream s1, and s2.wait_event(event) makes all future work on s2 wait for it without blocking the CPU. s2.wait_stream(s1) is exactly those two calls. Host to device copies can overlap compute only when the source is in pinned (page-locked) host memory and the copy is called with non_blocking=True.

The memory hierarchy

Data lives at several levels, each larger and slower than the one before:

• Registers, private to each thread: the operands arithmetic reads directly.
• Shared memory and L1, on each SM and shared by a thread block: the scratchpad a matmul kernel stages tiles in.
• L2 cache, shared by all SMs.
• Device memory (HBM or GDDR): the gigabytes that hold parameters, activations and optimizer state. "GPU memory" in nvidia-smi and out-of-memory errors means this level.
• Host memory, reached over PCIe or a faster link, far slower again.

A kernel's inputs start in device memory and its outputs end there, so every kernel pays at least one read of its inputs and one write of its outputs at device-memory bandwidth. What happens in between, in registers and shared memory, is nearly free by comparison. PyTorch's caching allocator keeps freed device memory reserved for reuse, which is why reserved memory stays high after tensors are deleted.

Arithmetic intensity and the roofline

An operation does some number of floating-point operations and moves some number of bytes to and from device memory. Its arithmetic intensity is the ratio. The time it needs is at least the larger of the time to do the arithmetic at peak and the time to move the bytes at full bandwidth:

intensity I = FLOPs / bytes moved     time ≥ max(FLOPs / peak, bytes / bandwidth)     attainable FLOP/s = min(peak, I × bandwidth)

Plotting attainable FLOP/s against intensity on log axes gives the roofline: a slanted line (bandwidth-limited) meeting a flat one (compute-limited) at the ridge point, I = peak / bandwidth. Operations left of the ridge are memory-bound and speed up only by moving fewer bytes; operations right of it are compute-bound and speed up only with faster or lower-precision arithmetic. The numbers on this page use an example GPU with a peak of 100 TFLOP/s and 2 TB/s of bandwidth, a ridge at 50 FLOP/byte. Real GPUs have ridges of the same order; check your own card's datasheet for its figures, which differ by precision.

Everything left of the ridge runs at intensity × bandwidth, however fast the arithmetic units are. The matmul is the only one of the four that can use the peak.

Why matmul is fast and elementwise ops are memory-bound

An elementwise op over N fp32 values reads 4N bytes, writes 4N, and does about N FLOPs: intensity 1/8, far left of any ridge, whatever N is. A matrix multiplication of two n × n matrices does 2n³ FLOPs on only 3n² values, so its intensity grows with n: each input value is used n times. The kernel turns that reuse into fewer bytes by tiling: a thread block loads a T × T tile of each input into shared memory once and uses every loaded value T times before loading the next tile. Without tiling every multiply-add would read both operands from device memory; with tiles of side T, device-memory traffic drops by a factor of T.

device-memory reads ≈ 2n³ / T values     intensity ≈ 2n³ / (2n³ / T × bytes per value) = T / bytes per value

Dedicated matrix units (tensor cores) raise the compute roof further, for lower-precision inputs, which pushes the ridge to the right and makes the gap between matmuls and everything else wider still. That is the subject of mixed precision, the next page.

Kernel fusion

If a memory-bound op's time is its bytes, the way to speed up a chain of them is to stop writing intermediate results to device memory. Kernel fusion merges several ops into one kernel that reads the inputs once, keeps intermediates in registers, and writes only the final output. A chain of k elementwise ops over the same tensor goes from 2k tensor-sized transfers to 2, and from k launches to one. The same idea at a larger scale is why fused attention kernels exist: they never write the full attention-score matrix to device memory. Fusion cannot help a compute-bound op; the matmul already reads each byte only a few times.

torch.compile in one paragraph

torch.compile(model) captures the Python code of a model into a graph (TorchDynamo), hands the graph to a backend ("inductor" is the default), and the backend generates fused kernels for the elementwise and reduction parts while calling vendor libraries for matmuls and convolutions. The first call is slow (compilation); later calls with the same shapes reuse the compiled code, and new shapes or Python-level changes the graph guards on trigger a recompile. mode="reduce-overhead" also records the kernels into CUDA graphs, replaying a whole sequence with one launch, which attacks the launch-bound case in the first diagram; mode="max-autotune" tries several matmul and convolution kernels and keeps the fastest. fullgraph=True makes any part the compiler cannot capture an error instead of a silent fall-back to eager Python.

The PyTorch API

Worked example

Implementation

import torch

x = torch.randn(4096, 4096, device="cuda")
w = torch.randn(4096, 4096, device="cuda")

# Timing GPU work: warm up, then measure with events (or synchronize around a CPU timer).
for _ in range(3):
    y = x @ w
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
for _ in range(10):
    y = x @ w
end.record()
end.synchronize()                       # wait until the work the event captured is done
ms = start.elapsed_time(end) / 10
tflops = 2 * 4096**3 / (ms / 1e3) / 1e12

# Overlapping a host-to-device copy with compute on a side stream.
batch = torch.randn(256, 4096).pin_memory()
copy_stream = torch.cuda.Stream()
with torch.cuda.stream(copy_stream):
    next_batch = batch.to("cuda", non_blocking=True)
y = x @ w                               # runs on the default stream meanwhile
torch.cuda.current_stream().wait_stream(copy_stream)
next_batch.record_stream(torch.cuda.current_stream())

# Fusing elementwise chains.
def gelu_bias(x, b):
    return torch.nn.functional.gelu(x * 0.5 + b)

fast = torch.compile(gelu_bias)         # first call compiles; later calls run one fused kernel

record_stream tells PyTorch's caching allocator that next_batch, allocated on copy_stream, is used on another stream, so its memory is not reused before that stream is done with it. The warmup loop matters: the first calls pay for library initialization, kernel selection and, under torch.compile, compilation.

Tradeoffs

• Batch size. Bigger batches raise the intensity of every weight-reading op (each weight is reused per row), moving matmuls from memory-bound to compute-bound, at the cost of memory and, in serving, latency.
• Fusion versus flexibility. Fused kernels (hand-written or compiled) remove traffic and launches, but compilation takes time, recompiles on new shapes, and is harder to debug than eager code.
• CUDA graphs. Replaying a recorded sequence removes launch overhead but needs static shapes and memory addresses, and holds that memory for as long as the graph lives.
• Extra streams. Overlapping copies or communication with compute hides their time, but adds ordering bugs (reading a tensor before its producer finished) that the default stream never has.
• Synchronization. Each .item() or CPU copy costs a pipeline drain; checking values every step is convenient for debugging and slow in production. Log a running tensor and read it every few hundred steps.

Connections

References

• Williams, Waterman and Patterson. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM 52(4), 2009.
• Dao, Fu, Ermon, Rudra and Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS, 2022. arXiv:2205.14135.
• Ansel et al. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. ASPLOS, 2024 (TorchDynamo and TorchInductor).
• NVIDIA. CUDA C++ Programming Guide: the programming model, memory hierarchy and streams chapters.
• PyTorch v2.15.0: torch/cuda/streams.py, torch.cuda.synchronize, torch.compile.