Activation functions

Problem

The neurons and layers page showed that stacking linear layers without anything between them collapses to one linear layer. Something nonlinear has to sit between them. But the choice matters in practice: it decides how well gradients travel backwards through many layers (see backpropagation), how fast training is, and which units can stop learning altogether.

Intuition

An activation is a small fixed curve applied to each number independently. Two properties of the curve decide almost everything:

• Its range and flatness. Sigmoid and tanh squash large inputs into a bounded interval and go flat there (saturate), so the gradient through them shrinks towards zero. ReLU and its relatives are unbounded on the positive side with slope 1, so the gradient passes through unchanged.
• What happens for negative inputs. ReLU outputs exactly 0 with slope 0 there: a unit whose input is always negative has no gradient and never recovers (a dead unit). Leaky ReLU, GELU and SiLU keep a small slope or a small negative dip, so something still flows back.

Modern transformers use smooth ReLU-like curves (GELU, SiLU) inside the feed-forward blocks, often as the gate of a gated linear unit (GLU) such as SwiGLU.

GELU and SiLU hug ReLU for positive inputs and are smooth at zero. Their dip below zero is small and shallow.

Mechanics

The functions and their slopes

Sigmoid's slope never exceeds 0.25 and tanh's never exceeds 1; both fall to zero away from the origin.

Saturation and vanishing gradients

Backpropagation multiplies the activation's slope into the gradient at every layer. If the slope is at most 0.25, as for sigmoid, ten layers shrink the signal by at least a factor 4¹⁰ ≈ one million. This is the vanishing gradient problem that kept deep sigmoid networks hard to train until ReLU and good initialization arrived. The cure is an activation whose slope is near 1 over a wide range, plus residual connections and normalization (see layer norm and residuals).

Dead units

ReLU's slope is exactly 0 for negative inputs. If a large learning-rate step pushes a unit's bias or weights so that its pre-activation is negative for every example, that unit's output and gradient are both 0 from then on. Nothing in the loss can bring it back. Leaky ReLU (a small fixed slope) or the smooth GELU and SiLU leave a path for gradient.

GELU and SiLU

GELU multiplies x by the probability Φ(x) that a standard normal variable is below x, a smooth gate that is nearly 0 for very negative inputs and nearly 1 for large positive ones. SiLU does the same with the logistic function σ(x) as the gate. Both are non-monotonic: they dip slightly below zero before rising. PyTorch's nn.GELU takes approximate="none" (exact, using the error function) or "tanh", the cheaper formula used in the original GPT-2 and BERT code.

Gated linear units

A GLU splits a projection into two halves a and b and returns a ⊙ gate(b). With a sigmoid gate that is the original GLU (PyTorch's F.glu); with SiLU as the gate on a it is SwiGLU, and with GELU it is GeGLU. The feed-forward blocks of many recent language models use SwiGLU: three weight matrices instead of two, with the hidden size reduced (typically to 2/3) to keep the parameter count similar. See the transformer page for where the feed-forward block sits.

Worked example

Implementation

import torch
import torch.nn.functional as F
from torch import nn

x = torch.linspace(-4, 4, 9)
print(F.relu(x), F.gelu(x), F.silu(x), F.gelu(x, approximate="tanh"))

class SwiGLUFeedForward(nn.Module):
    def __init__(self, d_model, d_hidden):
        super().__init__()
        self.gate = nn.Linear(d_model, d_hidden, bias=False)
        self.up = nn.Linear(d_model, d_hidden, bias=False)
        self.down = nn.Linear(d_hidden, d_model, bias=False)

    def forward(self, x):
        return self.down(F.silu(self.gate(x)) * self.up(x))

F.glu takes one tensor and splits it in half; the SwiGLU feed-forward above uses two separate projections instead, which is how most transformer implementations write it.

Tradeoffs

• ReLU is cheap and can die. Its simplicity is hard to beat for small models, but a bad learning rate can leave units permanently off.
• Smooth curves cost more. GELU and SiLU need an exponential or error function per element, though on a GPU the cost is small next to the matrix multiplications.
• Sigmoid and tanh are for gates and outputs. They are still right for recurrent-network gates, binary probabilities and bounded outputs, but not as the default hidden activation of a deep network.
• Gated units add parameters. SwiGLU uses three matrices where a plain feed-forward uses two, so the hidden size is usually adjusted for a fair comparison.
• Evidence is empirical. The gains of GELU, SiLU and SwiGLU over ReLU are measured on benchmarks, not proved; the difference between the smooth variants is small.
• Exact and approximate GELU differ. Match the variant a checkpoint was trained with.

Check yourself

Connections

References

• Hendrycks and Gimpel. Gaussian Error Linear Units (GELUs). 2016. arXiv:1606.08415.
• Ramachandran, Zoph and Le. Searching for activation functions. 2017. arXiv:1710.05941.
• Dauphin, Fan, Auli and Grangier. Language modeling with gated convolutional networks. ICML 2017. arXiv:1612.08083.
• Shazeer. GLU variants improve transformer. 2020. arXiv:2002.05202.
• Glorot, Bordes and Bengio. Deep sparse rectifier neural networks. AISTATS 2011.
• PyTorch v2.15.0: GELU, F.glu.