Latency and throughput

Problem

You deploy an 8B-parameter model in bf16 on one accelerator with 989 TFLOP/s of dense bf16 compute, 3.35 TB/s of memory bandwidth and 80 GB of memory (round numbers for an H100-class part). Product asks two questions. "How fast is it?" and "How many users can it take?" The first has at least three answers, and each one moves differently: how long until the first word appears, how fast words stream after that, and how long the whole answer takes. The second depends on the first, because you can almost always serve more users by making each one wait longer.

Without a model of where the time goes, people tune the wrong thing: buying FLOPs for a workload limited by bandwidth, reporting a mean that hides a tail of timeouts, or running a server at 95% utilization and wondering why p99 tripled. The model-serving page built the batching machinery; this page explains the numbers it produces.

Intuition

Generating text has two phases. Prefill runs the whole prompt through the model at once: 1,024 tokens multiply against every weight, so the GPU does a lot of arithmetic per byte of weight it reads. Decode then produces one token per step, and each step has to read all 16 GB of weights plus the KV cache to do only two FLOPs per weight per sequence. A GPU can do about 295 FLOPs in the time it reads one byte, so a single decode stream uses well under 1% of the compute. Decode is a bandwidth problem.

That is why batching works so well for decoding: one read of the weights serves every sequence in the batch. Tokens per second rise almost linearly with batch size while each step barely slows down, until the KV cache, which is per-sequence and grows with context, starts to dominate the bytes read. Batching is how you buy throughput, and each user pays for it in latency.

Mechanics

The latency metrics

• Time to first token (TTFT): from request arrival to the first output token. It is queue wait plus prefill. It is what makes a chat feel responsive.
• Inter-token latency (ITL), also called time per output token: the gap between consecutive tokens, which is one decode step of the batch the request is in. Reading speed is about 5–10 tokens per second, so an ITL under ~100 ms streams faster than anyone reads.
• End-to-end latency: TTFT + (n − 1)·ITL for n output tokens. It matters when another program waits for the whole answer, as in agents and batch jobs.
• Throughput: output tokens per second across all users (or requests per second). Goodput is throughput counting only requests that met their latency target.

The roofline

A kernel that does F FLOPs and moves M bytes takes at least

t = max(F / peak FLOP/s, M / bandwidth)

Its arithmetic intensity I = F/M decides which term wins: below the ridge point peak/bandwidth = 295 FLOP/byte it is memory-bound, above it compute-bound. For one decode step at batch B with context L:

F = B·(2·P + 4·layers·d·L),   M = 2·P + B·L·(KV bytes per token)

where P = 8B parameters at 2 bytes each, and each token of cache costs 2 (K and V) × 32 layers × 8 KV heads × 128 × 2 bytes = 131,072 bytes (the KV-cache page's configuration, with grouped-query attention). With no cache, I ≈ B, so the weights alone would need a batch near 295 to become compute-bound. The cache term grows with B too, so at long context the step never gets there.

Throughput against latency

Holding the weights fixed, the step time is roughly a + b·B: a constant for reading the weights plus a slope for each sequence's cache. Throughput is B/(a + b·B), which rises toward 1/b; latency per token rises linearly. A server picks a point on that curve, usually by capping the batch so ITL stays under a target, and continuous batching keeps the batch full as requests finish. Two hard limits cap B: memory (weights plus every sequence's cache must fit) and the compute ridge. Prefill tokens share the same GPU, so each prompt that arrives stalls everyone's decoding for its prefill time unless the server splits prefills into chunks.

Little's law

For any stable system, averaged over a long enough window,

L = λ · W

the mean number of requests in the system equals the arrival rate times the mean time each spends in it. It needs no assumption about arrival distributions or service order. For an LLM server, L is the number of sequences being decoded, which is the batch size. So a target rate fixes the batch, and the batch fixes the ITL, which feeds back into W. If the loop has no solution below the memory limit, the queue grows without bound.

Tail latency

Latency is a distribution, and the tail grows faster than the mean as utilization ρ approaches 1. For a single server with random (Poisson) arrivals and a fixed service time s, the mean time in system is

W = s + ρ·s / (2(1 − ρ))

(Pollaczek–Khinchine), which blows up as ρ → 1, and the high percentiles blow up sooner. Tails also compound: a request that fans out to k backends waits for the slowest, so if each is slow with probability p, the request is slow with probability 1 − (1 − p)^k. Report p50, p90 and p99, never a mean alone.

Measuring it in PyTorch

CUDA kernels run asynchronously on a stream, so a Python timer around model(x) measures how long it took to queue the work. Time GPU work with torch.cuda.Event(enable_timing=True): record one before and one after on the stream, synchronize, then read elapsed_time in milliseconds. To see which kernels the time goes to, use torch.profiler.profile with a schedule that skips warmup steps, call step() per iteration, and read key_averages() or export_chrome_trace. with_flops=True estimates FLOPs for matmuls and convolutions, which with the kernel time gives the achieved TFLOP/s to put on a roofline.

Worked example

The model and accelerator from the problem, with every decoding sequence holding 2,048 tokens of context, 1,024-token prompts and 256-token answers. These are roofline lower bounds (real kernels reach perhaps 60–80% of them, and activations and framework overhead are ignored), computed by tested helpers in ml_models_docs.py.

Implementation

Measuring TTFT and ITL for one request with CUDA events, then profiling a few decode steps:

import torch
from torch.profiler import profile, schedule, ProfilerActivity

@torch.inference_mode()
def time_request(model, prompt_ids, n_new):
    start = torch.cuda.Event(enable_timing=True)
    marks = [torch.cuda.Event(enable_timing=True) for _ in range(n_new)]
    start.record()
    logits, cache = model(prompt_ids, cache=None)          # prefill
    tok = logits[:, -1].argmax(-1, keepdim=True)
    marks[0].record()                                      # first token ready
    for i in range(1, n_new):
        logits, cache = model(tok, cache=cache)            # one decode step
        tok = logits[:, -1].argmax(-1, keepdim=True)
        marks[i].record()
    marks[-1].synchronize()                                # wait for the GPU
    ttft = start.elapsed_time(marks[0])
    itl = [a.elapsed_time(b) for a, b in zip(marks, marks[1:])]
    return ttft, sorted(itl)[len(itl) // 2]                # ms, median ITL

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
             schedule=schedule(wait=1, warmup=2, active=3), with_flops=True) as prof:
    for _ in range(6):
        time_request(model, prompt_ids, n_new=32)
        prof.step()
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

The first calls include CUDA context setup, allocator growth and kernel autotuning, which is why the schedule waits and warms up before recording. For a server, measure latency at the client, including queueing and network, and report percentiles over a load test at a fixed arrival rate. A closed loop where each client waits for its answer before sending again hides queueing, because a slow server slows its own load.

Tradeoffs

• Batch size is the main dial. Larger batches buy throughput (cost per token) and charge every user ITL. Set it from an ITL or p99 target, not from what fits in memory.
• Prefill and decode fight. A long prompt arriving stalls every running decode for its prefill. Chunked prefill bounds the stall; separate prefill and decode machines remove it at the cost of moving the KV cache between them.
• Memory bandwidth buys decode speed; FLOPs buy prefill. Weight and cache quantization help decode almost proportionally because they cut bytes read. Speculative decoding spends idle compute to emit several tokens per weight read.
• Little's law is exact but averaged. It gives the mean concurrency, not the peak. Bursts need headroom above λ·W.
• Utilization and tail latency trade directly. Running hot saves hardware and spends p99. Shedding or queueing load at the edge keeps the served requests within target.
• The roofline is a bound, not a prediction. Small kernels, launch overhead (see GPU execution) and attention's irregular memory access keep real numbers below it; the profiler tells you how far.

Connections

References

• Little. A Proof for the Queuing Formula: L = λW. Operations Research, 1961.
• Williams, Waterman and Patterson. Roofline: An Insightful Visual Performance Model for Multicore Architectures. CACM, 2009.
• Dean and Barroso. The Tail at Scale. CACM, 2013.
• Pope et al. Efficiently Scaling Transformer Inference. MLSys, 2023.
• Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP, 2023.
• Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (chunked prefill). OSDI, 2024.
• PyTorch v2.15.0: torch.cuda.Event, torch.profiler.profile, schedule.