Neural Networks
Advanced 16 min read Neural Networks

Training Pitfalls & Failure Modes

Symptom → cause → fix for the failures that actually happen

Networks fail in recognizable patterns: NaN losses, plateaued plateaus, silent overfitting, degenerate predictions. This concept is your symptom → cause → fix playbook.

NaN lossdead reluvanishing gradientsplateauoverfittingfailure modesdebugging
Prerequisites: Gradient Descent Initialization & Normalization
Before we start

What you will learn

  • Diagnose the five most common training failures by signature
  • Map each to its root cause and the cheapest fix
  • Add observability that finds these before humans do
Introduction

What is this?

Training fails in a handful of reproducible signatures: loss → NaN (exploding gradients / unstable ops), loss stuck (dead ReLUs, saturating activations, or a silent LR=0), deep stack underperforming shallow baseline (vanishing gradients), loss falling on train while val is flat (overfitting), and constant/degenerate outputs (post-softmax degeneracy or label collapse). Recognizing the signature tells you which fix applies — before you rewrite the architecture.

Motivation

Why do you need it?

  • Most 'my model is broken' sessions are one of a half-dozen known traps.
  • Each signature has a cheapest, most likely fix — order matters (LR before architecture).
  • Observability (grad norms, activation stats, weight norms) surfaces the failure before a bad model ships.
  • Knowing the shape of each failure lets you verify the fix fast instead of guessing.
Think about it

Mental model

Imagine

A doctor reading vital signs

Think of

Diagnosing training failures with signals

Fever, low BP, and a racing pulse point to different causes even when the patient 'felt sick' in one way. Train loss is just one vital; weight-norm, grad-norm and activation stats are the others that separate the diagnoses.

Mechanics

How it works

  • NaN: any unstable math multiplies up (exp, log on extreme values) or LR spikes a weight norm into overflow.
  • Dead ReLU: high LR/init permanently zeroes many units; gradients through them vanish.
  • Vanishing: saturating activations + depth keep early-layer gradients near zero.
  • Overfitting: capacity ≫ data with no regularization; val flattens while train falls.
  • Degenerate output: label imbalance + naive loss, or a collapsed embedding cluster, produces one class to rule them all.

Observed signals and the failure they point to.

NaN
Flat loss
Val flat
Deep < shallow
One-class
Real world

In production

  • CI gates run a short overfit test so plumbing failures fail before expensive fine-tuning.
  • Run-time dashboards plot grad-norm and weight-norm along with loss
  • Failsafe policy: NaN loss triggers automatic restart from last-good checkpoint with LR halved.
Guidelines

Best practices

  • Instrument grad-norm, weight-norm, and activation mean/std every K steps.
  • Run the LR range test once per architecture; log the chosen peak LR.
  • Gate training on a finite-loss assertion at step 1.
  • Cross-validate on a frozen split; never tune on the test set.
Watch out

Common mistakes

  • Rewriting the architecture when a 10x LR change and gradient clipping would have fixed it.
  • Adding regularization one knob at a time into an overfit mess instead of the right one first.
  • Trusting a smooth loss curve as evidence of a good model.
  • Ignoring class imbalance until the confusion matrix (post-deployment) reveals it.
Root cause analysis

Common issues

Error
Loss prints NaN by epoch 2
Root cause

LR too high, log/exp on near-zero values, or fp16 overflow

Fix

Drop LR 10x, add `1e-9` clamps to logs, enable mixed-precision autocast (bf16).

Error
All ReLUs dead — every hidden unit outputs 0
Root cause

Init too wide or LR too large permanently zeroed pre-activations

Fix

Kaiming init + lower LR; or switch to Leaky ReLU.

Error
Deep net matches a linear baseline exactly
Root cause

Vanishing gradients across depth (sigmoid chain) or degenerate init collapse

Fix

ReLU/GELU, residual connections, batchnorm/layernorm, proper initialization.

When things break

Debugging tips

  • NaN at step K? Run verbose for the K-first batches and print values at the first break spot.
  • Flat loss? Check LR isn't 0, then activation saturation, then data ranges.
  • Overfit? Compare loss-of-batch vs whole-run: see if it's validation curve shape, not just the number.
  • Print preds distribution: if one class always wins, the loss/target alignment is off, or imbalance.
Interviews

Interview checkpoint

Practice

Hands-on lab

Induce and Fix Three Failure Modes
30 min · advanced
In progress
Objectives
  • Reproduce exploding gradiendts, dead ReLU, and overfit gaps in controlled runs
  • Apply the correct fix and verify the signature changes
  • Build a simple 'checkpoint on NaN' safeguard

Induce and Fix Three Failure Modes

30 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Your train loss descends but validation stays flat. What is the most likely diagnosis?

2

Gradient clipping alone fixes all NaN-loss causes.

Go deeper

Official documentation

A Recipe for Training Neural Networks (karpathy)
docs