Neural Networks
Intermediate 14 min read Neural Networks

Forward Propagation

The pass that turns inputs into predictions

Forward propagation pushes inputs through the layers, producing predictions and cached intermediates. It's the pass that computes outputs — and everything backprop needs.

forward passforward propagationlogitsinferencematrix multiplycaching
Prerequisites: Multi-Layer Perceptron
Before we start

What you will learn

  • Trace a batch through an MLP layer by layer
  • Explain what the forward pass caches and why backprop reuses it
  • Count the FLOPs and memory of one forward pass
Introduction

What is this?

Forward propagation evaluates a network on an input: for each layer, multiply by weights, add bias, apply activation, feed the result to the next layer, until logits and a loss emerge. During training it also caches the activation of every layer, because backprop's chain rule needs those intermediates. At inference it's the entire prediction path.

Motivation

Why do you need it?

  • It defines the full input → output function a model learned.
  • Caching forward intermediates is what makes backprop ~2x a forward pass, not Nx.
  • Batch algebra (matmul) is the prime optimization target for GPU speedups.
  • Debugging 'wrong output' usually means tracing this pass, not the backward one.
Think about it

Mental model

Imagine

A lighthouse relaying light through lenses

Think of

Forward propagation through layers

Light enters the bottom lens, is refracted and focused at each stage, and emerges as the visible beam. Replace light with numeric signals and each lens with a weight+activation, and you've described the forward pass.

Mechanics

How it works

  • Input batch, shape (N, d0), enters layer 1; z1 = X @ W1 + b1 has shape (N, h1).
  • Activation h1 = f(z1) adds nonlinearity; this h1 is cached for backprop.
  • Repeat through hidden layers; the final layer outputs logits (N, C).
  • Softmax/sigmoid converts logits to probabilities; loss scores them.
  • Time cost ≈ sum over layers of 2·N·h_prev·h_cur multiply-adds; memory ≈ activations cached.

One batch through three layers.

X (N×d)
z1 = XW1+b1
h1 = relu(z1)
z2 = h1W2+b2
logits / probs
Real world

In production

  • Serving runs the forward pass only: no backward graph, single pass with cached weights at inference.
  • GPU throughput optimization batches forward passes (increase batch size) to hide latency.
  • Quantization/pruning rules are validated by running the forward pass on benchmark sets.
Guidelines

Best practices

  • Write inference forward passes with torch.no_grad() to skip building autograd graphs.
  • Run the forward pass on a single sample to verify shapes before scaling the batch.
  • Cache activations only during training; never during inference.
  • Profile the matmul stages — they dominate; everything else is overhead.
Watch out

Common mistakes

  • Building the graph in inference and paying memory/CPU for gradients you won't take.
  • Mismatching input dimensions (batch, features) at layer boundaries — the classic rank-1 vs rank-2 trap.
  • Applying the loss before the final nonlinearity (softmax) and printing 'probabilities' out of range.
  • Forgetting Transposed-weight orientation when hand-writing affine (W.T vs W).
Root cause analysis

Common issues

Error
Forward pass output is shape (N,1) but targets are (N,)
Root cause

Dimension conventions between logits and labels mismatch

Fix

Align shapes (expand to (N,1) or squeeze), then re-check the loss input.

Error
Activation values look huge after the second layer
Root cause

Unnormalized inputs or weights too large propagate through matmul

Fix

Normalize inputs, use good init, and add batchnorm/proper scaling.

Error
Inference is slower than training step
Root cause

Autograd graph being built during 'inference'

Fix

Wrap inference in torch.no_grad() (or your framework's eval mode).

When things break

Debugging tips

  • Print each layer's z and h shapes once on a dummy batch; shape errors are the most common forward bug.
  • Assert activations are finite after layer 1 — NaN often appears before loss, not after.
  • Compare model(x) to a hand-rolled matmul on 2 samples for spelling-safe verification.
  • In PyTorch, check requires_grad only on train-time tensors and use plain tensors at relative inference.
Interviews

Interview checkpoint

Practice

Hands-on lab

Trace a Forward Pass by Hand
25 min · intermediate
In progress
Objectives
  • Compute an MLP forward pass in numpy with explicit shapes
  • Verify against PyTorch (or reference) speed sizes
  • Catch the classic shape bugs that only appear at boundaries

Trace a Forward Pass by Hand

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which of these does the forward pass NOT produce?

2

The forward pass is run both for training and for inference.

Go deeper

Official documentation

PyTorch nn.Sequential forward docs
docs