Activation Functions
The nonlinear hearts of every layer
ReLU, sigmoid, tanh, softmax — activations shape what a layer can represent and how gradients flow. Choosing wrong means vanishing gradients, dead neurons or unstable training.
What you will learn
- Compare the standard activation functions
- Explain vanishing gradients and dead ReLUs
- Pick activations by layer role (hidden vs output)
What is this?
Activations introduce the nonlinearity neural networks need. In hidden layers, ReLU (and friends like GELU) sparsify and stabilize gradients; at the head, softmax turns logits into probabilities and sigmoid handles binary decisions. Their properties — gradient magnitude, saturation, zero regions — directly drive training health.
Why do you need it?
- Sigmoid saturates and kills gradients in deep stacks, which is why it faded from hidden layers.
- Softmax's winner-take-all temperament is exactly right for classification heads.
- GELU/SiLU smoothness matters for modern architectures' stability.
- Wrong activation = slow, unstable or dead training that no hyperparameter search fixes.
Mental model
Doors of different sizes
Activation functions
A fire door lets everything through unless fire (very negative) arrives (hard door); a turnstile lets a rate through (proportional); a gate blocks nothing dangerous but lets everything (leaky). The door's shape sets what energy flows to the next room.
How it works
- ReLU(x)=max(0,x): fast, sparsifying, zero-gradient for negatives (dead unit risk).
- Leaky/PReLU keep a small negative slope to avoid dead units.
- Sigmoid=1/(1+e^-x): between 0 and 1, saturating — good at binary/slow heads, bad deep.
- Tanh: zero-centered, stronger gradients than sigmoid, still saturating.
- Softmax: normalizes logits to probabilities summing to 1 — the classification head.
- GELU/SiLU: smooth approximations of ReLU, common in transformer feed-forward layers.
Choosing activations by role.
In production
- Transformers use GELU in feed-forward layers for training stability at scale.
- A binary safety head outputs sigmoid; the 0..1 score feeds threshold logic.
- Multiclass intent routing pushes logits through softmax for clean probabilities.
Best practices
- Default to ReLU-family in hidden layers; reach for GELU when training at transformer scale.
- Use sigmoid for binary probabilities and softmax for mutually-exclusive classes.
- Watch for dead-ReLU (zero-ish gradient mass) early in training.
Common mistakes
- Putting tanh/sigmoid in deep hidden layers and wondering why training stalls.
- Feeding raw logits into a downstream system expecting probabilities.
- Ignoring that softmax is over classes, not examples.
Common issues
Training loss flatlines earlyVanishing gradients from saturating activations or bad init
Switch to ReLU-family, improve normalization/init, and check gradient norms.
Many neurons never change after trainingDead ReLUs with zero gradient on negatives
Consider LeakyReLU/PReLU or lower learning rate.
Debugging tips
- Gradient norm ~0 after warm-up — check for saturating activations.
- A layer output locked at 0 for whole batches — those ReLUs are dead; consider leaky variants.
- Probabilities all equal after softmax — logits are near-zero or uniform; usually an init problem.
Interview checkpoint
Hands-on lab
- Implement sigmoid, tanh and ReLU in pure Python
- Numerically show where each function saturates and loses gradient
- Pick an activation for a given network depth
Activations and the Vanishing-Gradient Trap
25 min · beginner
Quick quiz
Which activation is the standard classification head?
Sigmoid is ideal for deep hidden layers because it is bounded.