Deep Learning
Intermediate 14 min read Deep Learning

Activation Functions

The nonlinear core of every layer

Activations inject the nonlinearity that lets stacked layers express complex functions. Choice of activation governs gradient flow, saturation and training stability.

activation functionReLUtanhsigmoidleaky ReLUnonlinearitydead neurons
Prerequisites: Deep Learning Basics
Before we start

What you will learn

  • Explain why nonlinearity is mandatory in stacked layers
  • Compare sigmoid, tanh, ReLU and variants by gradient behavior
  • Diagnose dead-ReLU and saturation from training signals
Introduction

What is this?

An activation function applies a fixed nonlinear transform to a layer's pre-activation (linear) output: ReLU zeroes negatives, sigmoid squashes to (0,1), tanh to (-1,1). Without nonlinearity, any stack of layers collapses into a single linear transformation — the network could express nothing more than linear regression. Activations are what let depth mean something.

Motivation

Why do you need it?

  • Two stacked linear layers equal one linear layer — nonlinearity is the reason depth buys capacity.
  • Sigmoid/tanh offer smooth, bounded outputs useful at the very end for probabilities.
  • ReLU avoids saturation in most of its range, keeping gradients alive through many layers.
  • Choosing badly (saturating activations deep) causes vanishing gradients and slow, stuck training.
Think about it

Mental model

Imagine

A dimmer switch vs a hard on/off switch

Think of

Sigmoid/tanh vs ReLU

A dimmer (sigmoid) is smooth but only changes output meaningfully in a narrow middle band — past the ends the 'wire' is saturated. A hard switch (ReLU) is blunt but always responsive: below cutoff it's off, above it passes everything through.

Mechanics

How it works

  • A layer computes z = Wx + b; the activation then applies element-wise: a = f(z).
  • ReLU: f(z) = max(0, z); linear for z>0, zero for z<=0 — cheap, no saturation above zero.
  • Leaky ReLU: max(0.01z, z) keeps a small gradient for negatives so dead neurons leak.
  • Sigmoid: 1/(1+e^-z), bounded (0,1); gradient maxes at 0.25 at z=0 and decays — a vanish risk in deep stacks.
  • tanh: bounded (-1,1), zero-centered, gradient maxes at 1 — a common choice before ReLU took over.

Activation shapes and their gradient ranges.

Linear z
ReLU
tanh
sigmoid
Next layer
Real world

In production

  • Image classifiers default to ReLU in convolutional stacks with a batchnorm before each block.
  • Output heads switch to sigmoid (binary logits) or softmax (multiclass) — the last activation is chosen for the loss.
  • Transformers use GELU; RNN/LSTM gates use sigmoid+tanh because bounded outputs keep gate ranges stable.
Guidelines

Best practices

  • Use ReLU-family activations in hidden layers; reserve sigmoid/tanh for output gates or final probabilities.
  • Put normalization before the activation (pre-activation) for smoother gradients.
  • Match the final activation to the loss (sigmoid+BCE, softmax+CE, none+MSE/linear head).
  • If many ReLUs are dead, try leaky ReLU or fix initialization instead of adding more units.
Watch out

Common mistakes

  • Using sigmoid throughout hidden layers — deep stacks suffer vanishing gradients.
  • Sigmoid-for-binary at the end but then passing through MSE and double-punishing confidence.
  • Counting dead neurons due to a wrong init (too-large weights zero everything) and 'fixing' it with width.
  • Assuming activations are interchangeable — GELU vs ReLU changes loss trajectories, not just speed.
Root cause analysis

Common issues

Error
The model outputs only one class after training
Root cause

Final sigmoid saturates and gradients vanish for the minority class

Fix

Use logits into BCEWithLogits (no final sigmoid) so the gradient stays finite near saturation.

Error
Most hidden activations are exactly 0
Root cause

Dead ReLUs from bad initialization or high learning rate permanently zeroing units

Fix

Lower init scale (Kaiming), reduce LR, or switch to leaky ReLU.

Error
Deep stack trains very slowly
Root cause

Saturating activations (sigmoid/tanh) without normalization

Fix

Swap to ReLU-family + batchnorm/groupnorm and re-init.

When things break

Debugging tips

  • Print the fraction of negative pre-activations: >50-60% dead-zone activation suggests dying layers.
  • Visualize activation histograms per layer after a few steps, not just loss curves.
  • If loss is flat near 0.69 (ln2) on binary, the end activation likely saturates.
  • A sudden NaN usually signals an activation like exp/log interaction, not the optimizer.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which activation's gradient is upper-bounded by 0.25?

2

A chain of layers with no activation function will behave exactly like a single linear layer.

Go deeper

Official documentation

PyTorch activation functions
docs