Quantization

Problem

A trained network stores every weight as a 32-bit or 16-bit float, but it rarely needs that much precision to give the same answers. Inference cost is dominated by moving bytes: during LLM decoding every weight is read from memory once per generated token (see the KV cache and how a GPU runs a model), so halving the bytes per weight nearly halves the time per token, and lets a model fit on a smaller device. Integer arithmetic is also cheaper than floating point on most hardware. Quantization replaces floats with small integers plus a few floats that say how to read them. The question is how to choose those few floats so the model's outputs barely change.

Intuition

An 8-bit integer has 256 values. Quantizing a tensor means laying a ruler with 256 evenly spaced marks over the range the tensor's values actually occupy, and storing each value as the number of its nearest mark. The spacing between marks is the scale; the mark that sits exactly on 0.0 is the zero point. Every value moves by at most half a spacing. So the whole game is keeping the ruler short: the narrower the range one scale has to cover, the finer its marks. That is why quantizers use a separate scale per output channel or per small group of weights, and why one huge value (an outlier) stretches the ruler and wipes out the precision of everything else that shares its scale.

Both rulers have 256 marks. The affine one is laid over exactly the range the values occupy; the symmetric one is centred on zero, so with mostly positive values almost half its codes go unused and its marks are 1.60× farther apart.

Mechanics

Scale and zero point

An integer type has a range [q_min, q_max]: [−128, 127] for torch.qint8, [0, 255] for torch.quint8. Affine (asymmetric) quantization maps a float x to an integer q and back with a scale s and an integer zero point z:

q = clamp(round(x / s) + z, q_min, q_max)     x̂ = (q − z) × s

The rounding is to the nearest integer with ties to even (PyTorch's kernels call std::nearbyint). Given the minimum and maximum a tensor takes, PyTorch's observers first stretch the range to include 0, then divide it evenly:

min⁻ = min(min, 0), max⁺ = max(max, 0)     s = (max⁺ − min⁻) / (q_max − q_min)     z = q_min − round(min⁻ / s)

Including 0 guarantees that 0.0 is represented exactly (it becomes z), which matters because padding, ReLU outputs and masked values are exact zeros. Symmetric quantization fixes z = 0 for signed types (128 for unsigned) and sets s = max(|min⁻|, max⁺) / ((q_max − q_min) / 2), which divides by 127.5 for qint8. The quantized matmul is then simpler (no zero-point cross terms), at the cost of wasting codes when the values are lopsided, as in the diagram. PyTorch's defaults follow that split: weights are symmetric qint8, activations affine quint8. Values outside the range are clamped, so the error per value is at most s/2 inside the range and unbounded outside it.

Granularity: per tensor, per channel, per group

One scale for a whole tensor (per_tensor_affine, per_tensor_symmetric) is cheapest but lets the largest value anywhere set the spacing for everything. A weight matrix's output channels (rows of a linear layer's weight) often differ in magnitude by an order of magnitude, so PyTorch's PerChannelMinMaxObserver keeps a separate min and max, and so a separate scale, along ch_axis (0 by default). Because each output channel's scale multiplies a whole output element, per-channel weight scales cost nothing in the matmul: the integer dot product is computed first and scaled once at the end. Weight-only LLM quantization goes further, with one scale per group of 32 to 128 consecutive weights within a row. Each fp16 scale over a group of g weights adds 16/g bits per weight. PyTorch's observer module names the same granularities as PerTensor, PerAxis, PerGroup, PerRow, PerToken and PerBlock.

Filled bars are the span of int8 codes each row actually uses; dashed outlines are the full range. Error is the row's total absolute rounding error relative to its total magnitude.

Post-training quantization

Post-training quantization (PTQ) quantizes a trained model without further training. In dynamic quantization the weights are quantized ahead of time and each activation's scale and zero point are computed from that activation at run time; quantize_dynamic's docstring calls this "dynamic (i.e. weights-only)", and by default it swaps nn.Linear and the recurrent layers for int8 versions (or fp16 with dtype=torch.float16). In static quantization the activation scales are fixed too: prepare inserts observers after each quantizable module, you run a sample of representative batches through the model (calibration), and convert replaces each module with a quantized one that uses the observed ranges. A MinMaxObserver tracks the running minimum and maximum; a MovingAverageMinMaxObserver averages them; a HistogramObserver chooses a range that minimizes quantization error, which can clip rare extreme values on purpose. Which observer each module gets comes from its QConfig: default_qconfig pairs a MinMaxObserver for activations (quint8, codes 0-127) with a symmetric qint8 per-tensor weight observer; default_per_channel_qconfig uses the per-channel weight observer instead.

Quantization-aware training

When PTQ loses too much accuracy, quantization-aware training (QAT) fine-tunes the model with the quantization error in the loop. prepare_qat inserts FakeQuantize modules that quantize and immediately dequantize, so values stay floats but take only the 256 representable levels:

x_out = (clamp(round(x / s + z), q_min, q_max) − z) × s

Rounding has zero derivative almost everywhere, which would stop all gradients. The fix is the straight-through estimator: treat rounding as the identity in the backward pass. PyTorch's kernel records a boolean mask in the forward pass, true where round(x/s) + z fell inside [q_min, q_max], and the backward pass is just dY * mask: the gradient passes through unchanged where the value was rounded and is zero where it was clamped. The weights learn to sit where rounding hurts least, and the observers inside each FakeQuantize keep updating the scales during training. After training, convert produces the real integer model.

Forward (green) sees quantized values; backward (amber) pretends rounding was the identity, except where clamping cut the value off.

Weight-only int8 and int4 for LLMs

For LLM decoding, which is memory-bound, the usual recipe is weight-only quantization: weights are stored in int8 or int4 and dequantized to fp16 or bf16 inside the matmul kernel, while activations stay in 16-bit. The speedup comes from reading fewer bytes, not from integer arithmetic. At 8 bits, symmetric per-channel round-to-nearest is usually close to lossless. At 4 bits there are only 16 levels, so round-to-nearest needs small groups, and better methods choose the rounding with data:

• GPTQ quantizes a layer's weights one column at a time and, after each column, adjusts the columns not yet quantized to compensate for the rounding error just introduced, using second-order information (the inverse Hessian of the layer's reconstruction error) computed from a small calibration set. It reaches 3 to 4 bits per weight on large models in a single pass, with no retraining.
• Outliers. In large transformers a few hidden dimensions carry activations far larger than the rest. With one scale per activation vector, a single outlier stretches the scale so far that the ordinary values round to zero (worked example below). LLM.int8() handles this with a mixed-precision decomposition: the hidden dimensions whose activations exceed a threshold (6.0 in the paper) are multiplied in fp16, and everything else in int8 with a separate scale per row of the activations and per column of the weights.

A note on the PyTorch API: in v2.15.0 the entry points in torch.ao.quantization's quantize.py (prepare, convert, quantize_dynamic, prepare_qat and the rest) are marked deprecated, with a warning that points to the separate torchao library (its quantize_ API for eager models, and prepare_pt2e and convert_pt2e for exported graphs). They still work, and the observers, fake quantization and the formulas above are the same ideas torchao builds on.

The PyTorch API

Worked example

Implementation

import torch
from torch.ao.quantization import (
    get_default_qconfig, prepare, convert, quantize_dynamic,
    get_default_qat_qconfig, prepare_qat,
)

# The primitives: a quantized tensor, its codes and its qparams.
t = torch.quantize_per_tensor_dynamic(torch.tensor([-1.0, 0.0, 1.0, 2.0]), torch.quint8, False)
t.int_repr()                       # tensor([  0,  85, 170, 255], dtype=torch.uint8)
t.q_scale(), t.q_zero_point()      # (0.011764705882352941, 85)

# Dynamic PTQ: int8 weights for every nn.Linear, activations quantized per batch.
qmodel = quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

# Static PTQ: observe, calibrate, convert.
model.eval()
model.qconfig = get_default_qconfig("x86")
prepared = prepare(model)          # inserts observers (model needs QuantStub/DeQuantStub)
with torch.no_grad():
    for x, _ in calibration_loader:
        prepared(x)                # observers record min/max
qmodel = convert(prepared)

# QAT: fake-quantize during fine-tuning, then convert.
model.train()
model.qconfig = get_default_qat_qconfig("x86")
qat = prepare_qat(model)
for x, y in loader:
    loss_fn(qat(x), y).backward()  # gradients flow through FakeQuantize via the STE
    optimizer.step(); optimizer.zero_grad()
qmodel = convert(qat.eval())

prepare, convert, quantize_dynamic and prepare_qat raise a deprecation warning in v2.15.0; new code should use torchao, which provides the same flows (and the int4 weight-only and GPTQ-style recipes for LLMs) on top of the same ideas. Eager-mode static quantization also needs QuantStub and DeQuantStub at the model's float/int boundaries and usually module fusion (conv + batch norm + ReLU) before prepare.

Tradeoffs

• Bits versus accuracy. int8 with per-channel weight scales is close to lossless for most models. int4 needs groups and, for the best results, data-aware rounding such as GPTQ; below 4 bits accuracy drops quickly.
• Weight-only versus weights and activations. Weight-only is simple and speeds up memory-bound decoding. Quantizing activations too unlocks integer matmuls for compute-bound work (large batches, prefill) but must handle activation outliers and needs calibration.
• Dynamic versus static. Dynamic quantization needs no calibration data but computes activation ranges on every call; static is faster at run time but only as good as the calibration set.
• PTQ versus QAT. PTQ takes minutes and no labels; QAT recovers more accuracy at low bit widths but costs a training run.
• Granularity. Finer scales (per channel, per group) cut error but add storage (16/g bits per weight) and kernel complexity; per-token activation scales help with outliers but must be computed on the fly.
• Range choice. Min/max ranges never clip but let one extreme value coarsen every other; histogram-based ranges clip a few values to keep the rest precise.

Connections

References

• Jacob, Kligys, Chen, Zhu, Tang, Howard, Adam and Kalenichenko. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. CVPR, 2018. arXiv:1712.05877 (the affine scheme with scale and zero point).
• Bengio, Léonard and Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. 2013. arXiv:1308.3432 (the straight-through estimator).
• Dettmers, Lewis, Belkada and Zettlemoyer. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS, 2022. arXiv:2208.07339.
• Frantar, Ashkboos, Hoefler and Alistarh. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR, 2023. arXiv:2210.17323.
• PyTorch v2.15.0: torch/ao/quantization/quantize.py, torch/ao/quantization/observer.py, torch/ao/quantization/fake_quantize.py.