Convolutional networks

Problem

A 224 × 224 colour image has 150,528 numbers. Feed it to a fully connected layer with even 200,704 outputs and you need about 30.2 billion weights in the first layer alone. Worse, those weights learn nothing that transfers across the image: a cat-ear detector in the top-left corner is a separate set of weights from the same detector one pixel to the right, so the network needs to see ears at every position to learn them everywhere.

Images have structure a dense layer ignores. Nearby pixels belong together, and the same pattern means the same thing wherever it appears. A good layer for images should look at small neighbourhoods and reuse its weights at every position.

Intuition

Take a 3 × 3 stencil of weights, a kernel. Lay it over the top-left corner of the image, multiply each weight by the pixel under it and add the nine products: that is one output number. Slide the stencil one pixel right and repeat. Sweep the whole image and you get a new grid of numbers, a feature map, that is large wherever the image locally looks like the kernel.

A layer has many kernels, each producing its own map, so the output is a stack of maps: channels. The next layer's kernels look at small neighbourhoods across all those channels, so they detect combinations of the first layer's patterns: edges become corners, corners become textures, textures become parts. Every layer sees a slightly larger region of the original image than the last, and downsampling (stride or pooling) makes that region grow quickly. By the end, each number summarises a large part of the image, and a small classifier on top reads off the answer.

Mechanics

Convolution as a sliding dot product

For one input channel and one kernel K of size k × k, output position (i, j) is

out[i, j] = bias + Σ_a Σ_b K[a, b] · x[i·s + a·d, j·s + b·d]

where s is the stride and d the dilation (both 1 in the plain case). With several input channels the sum also runs over every channel, so each kernel is a Cin × k × k block and each output channel has its own kernel and bias. Deep learning libraries compute this cross-correlation; the textbook convolution flips the kernel first, which makes no difference when the kernel is learned.

A Conv2d(Cin, Cout, k) therefore has a weight of shape [Cout, Cin/groups, k, k] and a bias of shape [Cout], and maps [N, Cin, H, W] to [N, Cout, Hout, Wout].

The kernel subtracts the right column of its window from the left one, so it responds only where bright meets dark: a vertical-edge detector. The flat regions give 0.

Stride, padding, dilation and the output size

• Stride s moves the window s pixels at a time, so the output shrinks by about a factor s.
• Padding p adds p rows and columns of zeros (or reflected or replicated values, padding_mode) around the input, so border pixels are covered as often as central ones and the size can be preserved. padding="same" computes it for you when the stride is 1.
• Dilation d spaces the kernel's taps d pixels apart: a 3 × 3 kernel with d = 2 covers a 5 × 5 area with nine weights.

Along each spatial dimension, PyTorch's Conv2d and MaxPool2d both give

H_out = ⌊(H_in + 2p − d·(k − 1) − 1) / s⌋ + 1

The floor means extra pixels that do not fill a final window are dropped. Pooling's ceil_mode=True rounds up instead, keeping a partial last window as long as it starts inside the input.

Channels, 1 × 1 convolutions and groups

Each output channel looks at every input channel, so a 3 × 3 convolution from 64 to 128 channels has 128 × 64 × 3 × 3 = 73,728 weights. A 1 × 1 convolution looks at one pixel but all channels: it is a linear layer applied at every position, used to change the channel count cheaply. groups=g splits the channels into g independent groups; with groups=Cin each channel gets its own kernel (a depthwise convolution), which with a following 1 × 1 convolution is the depthwise-separable block in MobileNet-style networks.

Pooling

Pooling replaces each window by a summary with no weights: MaxPool2d(2) keeps the largest value in each 2 × 2 block, AvgPool2d(2) the mean. The stride defaults to the kernel size, so windows do not overlap and each side halves. Max pooling makes the output insensitive to small shifts of a feature inside its window. AdaptiveAvgPool2d(1) averages each whole channel to one number whatever the input size, which is how modern CNNs go from feature maps to a vector for the classifier ("global average pooling").

Receptive fields

The receptive field of an output unit is the region of the input that can affect it. Track two numbers through the layers: the receptive field r and the jump j (how many input pixels one step in this layer's output corresponds to). Starting from r = 1, j = 1, each layer with kernel k, stride s and dilation d does

r ← r + (k − 1)·d·j    j ← j·s

So a 3 × 3 layer adds 2j pixels: depth grows the field linearly, and each stride multiplies how fast later layers grow it. Three stacked 3 × 3 convolutions see 7 × 7, the same as one 7 × 7, with 110,592 instead of 200,704 weights at 64 channels and two extra nonlinearities in between. That observation from VGG is why nearly every modern CNN uses 3 × 3 kernels. The theoretical field is an upper bound; the effective field, where the gradient is large, is smaller and roughly Gaussian.

Parameter sharing and translation equivariance

The same kernel is used at every position, so a convolution has Cout·Cin·k² weights no matter how big the image is. Sharing also makes the layer translation equivariant: shift the input and the feature map shifts by the same amount (exactly, for stride 1 away from the borders). Pooling and the final global average then make the prediction roughly invariant to where the object is. These are built-in assumptions, an inductive bias, which is why CNNs learn from far less data than a model that has to discover locality on its own.

Residual blocks

Stacking more plain convolution layers eventually makes training error go up, not down: the deeper network is harder to optimise even though it could copy the shallower one and add identity layers. ResNet fixes this by having each block learn a correction to its input:

y = ReLU(F(x) + shortcut(x))    F = conv3×3 → BN → ReLU → conv3×3 → BN

If F learns nothing, the block passes x straight through, so extra depth cannot hurt, and the gradient has a direct path back through every addition (see normalization and residuals, where the transformer uses the same trick). When a block halves the resolution and doubles the channels, x and F(x) no longer have the same shape, so the shortcut becomes a strided 1 × 1 convolution with batch norm (the "downsample" path). The convolutions before a batch norm have bias=False, since the norm's own shift makes a bias redundant.

The first block of ResNet-18's layer2. Its second block, and every block of layer1, keeps the shape and uses the identity shortcut.

Worked example

Implementation

A convolution in plain Python, the way the formula reads (the tested helper behind this page's numbers):

def conv2d(x, weight, bias=None, stride=1, padding=0, dilation=1):
    # x: [C_in][H][W], weight: [C_out][C_in][k][k]
    planes = [pad2d(plane, padding) for plane in x]
    k = len(weight[0][0])
    out_h = (len(x[0]) + 2 * padding - dilation * (k - 1) - 1) // stride + 1
    out_w = (len(x[0][0]) + 2 * padding - dilation * (k - 1) - 1) // stride + 1
    return [
        [
            [
                (bias[o] if bias else 0) + sum(
                    kernels[c][a][b] * planes[c][i * stride + a * dilation][j * stride + b * dilation]
                    for c in range(len(x)) for a in range(k) for b in range(k)
                )
                for j in range(out_w)
            ]
            for i in range(out_h)
        ]
        for o, kernels in enumerate(weight)
    ]

The same networks in PyTorch:

import torch
from torch import nn
import torch.nn.functional as F

lenet = nn.Sequential(
    nn.Conv2d(1, 6, 5), nn.ReLU(), nn.MaxPool2d(2),     # [N, 6, 14, 14]
    nn.Conv2d(6, 16, 5), nn.ReLU(), nn.MaxPool2d(2),    # [N, 16, 5, 5]
    nn.Flatten(),
    nn.Linear(400, 120), nn.ReLU(),
    nn.Linear(120, 84), nn.ReLU(),
    nn.Linear(84, 10),
)
print(sum(p.numel() for p in lenet.parameters()))       # 61706


class BasicBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.shortcut = nn.Identity()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels),
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + self.shortcut(x))


block = BasicBlock(64, 128, stride=2)
print(block(torch.zeros(1, 64, 56, 56)).shape)          # torch.Size([1, 128, 28, 28])

Stacking a 7 × 7 stride-2 stem, a 3 × 3 stride-2 max pool, four stages of two such blocks (64, 128, 256 and 512 channels), global average pooling and a 512 → 1000 linear layer gives exactly the 11,689,512 parameters in the table. Real libraries compute convolutions with im2col plus a matrix multiply, Winograd or FFT transforms, or cuDNN kernels picked per shape; the arithmetic is the same.

Tradeoffs

• Inductive bias. Locality and weight sharing make CNNs data-efficient and fast, but they also limit what a layer can relate: two distant pixels only interact many layers up. Vision transformers drop the bias, need more data or pretraining, and win at scale.
• Downsampling. Stride and pooling grow the receptive field cheaply and cut compute, but throw away exact position. Dense prediction (segmentation, detection) adds it back with skip connections, dilation or upsampling.
• Kernel size. Stacked 3 × 3 kernels give the same field as one large kernel with fewer parameters and more nonlinearity; recent ConvNets bring back 7 × 7 depthwise kernels because they are cheap when depthwise.
• Padding keeps sizes tidy but tells the network where the border is, which slightly breaks translation equivariance.
• Stride versus pooling. A strided convolution learns how to downsample; max pooling is free and adds a little shift invariance. Modern networks mostly use strided convolutions after the stem.
• Batch norm in residual blocks makes deep CNNs trainable at high learning rates but ties the output to batch statistics; small-batch training often switches to group norm.

Check yourself

Connections

References

• LeCun, Bottou, Bengio and Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 1998 (LeNet-5).
• Krizhevsky, Sutskever and Hinton. ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS, 2012 (AlexNet).
• Simonyan and Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR, 2015 (arXiv 1409.1556; stacked 3 × 3 kernels).
• He, Zhang, Ren and Sun. Deep Residual Learning for Image Recognition. CVPR, 2016 (arXiv 1512.03385).
• Dumoulin and Visin. A guide to convolution arithmetic for deep learning. 2016 (arXiv 1603.07285).
• Luo, Li, Urtasun and Zemel. Understanding the Effective Receptive Field in Deep Convolutional Neural Networks. NeurIPS, 2016 (arXiv 1701.04128).
• PyTorch v2.15.0: torch/nn/modules/conv.py (_ConvNd), Conv2d, torch/nn/modules/pooling.py.