Deep Learning
Intermediate 15 min read Deep Learning

Initialization & Normalization

Starting and keeping signals in a learnable range

How weights start and how intermediate values are scaled determines whether deep networks learn at all. Xavier/Kaiming init and batch/group normalization keep activations in a stable regime.

initializationXavierKaimingbatch normalizationlayer normtraining stability
Prerequisites: Backpropagation
Before we start

What you will learn

  • Choose initialization schemes that match activation functions
  • Explain what normalization layers fix and where to place them
  • Diagnose stability issues that read as 'loss stuck' or 'gradient collapse'
Introduction

What is this?

Initialization sets the first weight values before any training, and normalization rescales the signals inside the network during training. If weights start too large or too small, activations and gradients immediately blow up or vanish no matter the optimizer. Normalization layers re-center and re-scale layer outputs every step, so depth stops amplifying drift, enabling very deep nets.

Motivation

Why do you need it?

  • Zero-initializing all weights makes every neuron identical — symmetric, and training never breaks symmetry.
  • Variance control: with N inputs, output variance scales with N unless init divides by √(fan_in).
  • BatchNorm stabilizes the signal distribution per layer, letting you train deeper, faster, at higher LR.
  • Without these, your first-debug is 'why won't my deep net learn' — with them it's usually something else.
Think about it

Mental model

Imagine

Starting a race from a mark, and adjusting stride on hilly terrain

Think of

Initialization and normalization

Init is choosing the correct starting position so nobody begins a meter into a wall; normalization is pacing so steep gradients mid-race don't make all participants trip simultaneously.

Mechanics

How it works

  • Xavier: initialize uniform/normal with variance 1/(fan_in) — the right scale for sigmoid/tanh activations.
  • Kaiming/He: variance 2/(fan_in) — tuned to ReLU, which halves the effective signal.
  • BatchNorm: normalize per-channel across the batch to zero mean/unit variance, then learn scale/shift γ,β.
  • LayerNorm normalizes per-sample across features (used in transformers where samples are independent).
  • The normalized signal keeps gradients magnitude-stable across depth, so deep stacks converge.

Normalization keeps layer outputs from drifting as depth grows.

Init
Linear
Normalize
Activate
Next layer
Real world

In production

  • Image nets default to Kaiming-init + BatchNorm (pre-activation) with weight decay.
  • Transformers use LayerNorm + a scaled init (GPT small init) to keep residual streams well-conditioned.
  • GANs and style models rely on careful init and normalization schemes (adaptive/spectral variants) to stay stable.
Guidelines

Best practices

  • Match init to activation: Kaiming for ReLU-family, Xavier for tanh/sigmoid.
  • Put BatchNorm before the activation; keep it on and frozen consistently between train/eval.
  • For residual nets, init the last layer of residual branches near zero so the network starts as identity.
  • Beware BatchNorm's batch-size dependence — for small or dynamic batches prefer LayerNorm/GroupNorm.
Watch out

Common mistakes

  • Initializing everything to zeros — symmetry breaks learning entirely.
  • Using Kaiming with tanh (wrong variance target) and getting saturated starts.
  • Applying one global normalization scale across heterogeneous tasks (multi-task heads).
  • Training with BatchNorm in a batch of 4 and wondering why val is noisy.
Root cause analysis

Common issues

Error
Loss never moves from its initial random level
Root cause

Saturated init (tanh/sigmoid) with perfectly zero gradients at the start

Fix

Switch to Kaiming/He-style init or reduce the initial weight variance by 10x.

Error
Validation quality swings train-to-train
Root cause

BatchNorm stats shift with tiny batches or stochastic order

Fix

Use running-stat Freezing/eval mode, larger batch, or LayerNorm.

Error
First-layer activations are all identical
Root cause

Symmetric init or a dataloader feeding one repeated batch

Fix

Reseed and randomize init; then visually dump one batch.

When things break

Debugging tips

  • Run a few forward passes with random data: activation std should stay ~O(1), not 1e-4 or 1e4.
  • Print first-batch gradient norms per layer; outliers expose init or normalization mismatches.
  • Toggle normalization off for two steps and watch the same seed blow up — confirms it's doing work.
  • In transformers, check activations at residual output gates, where drift accumulates first.
Interviews

Interview checkpoint

Practice

Hands-on lab

Watch Init and Norm Affect Training
25 min · intermediate
In progress
Objectives
  • Compare zero, bad-scale and Kaiming initialization on the same training run
  • Observe how normalization changes the first-batch loss trajectory
  • Learn to diagnose saturation from activation statistics

Watch Init and Norm Affect Training

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which initialization scheme is designed for ReLU-family activations?

2

Batch normalization fully removes the need for good weight initialization.

Go deeper

Official documentation

BatchNorm paper
docs
PyTorch — normalization layers
docs