Neural Networks
Advanced 17 min read Neural Networks

MLP from Scratch

Removing the framework's magic to own the internals

Building a full MLP in numpy — init, forward, backward, optimizer, batching — turns framework magic into a checklist you can debug. This concept walks a complete training loop you can run in a sandbox.

MLP numpyfrom scratchtraining loopmanual backpropmini-batchtoy implementation
Prerequisites: Forward Propagation Backpropagation
Before we start

What you will learn

  • Implement init, forward and backward for a 2-layer net
  • Write a mini-batch loop with an optimizer step
  • Verify correctness through overfitting + gradient checks
Introduction

What is this?

A from-scratch MLP is every piece the framework usually hides: random weight init to a sane scale, matrix forward, per-layer gradient computation via the chain rule, an optimizer loop that applies updates per mini-batch, and a split that lets you overfit sanity-check. Writing it once converts vague 'neural net' talk into concrete, debuggable code.

Motivation

Why do you need it?

  • Framework-shaped debugging is powerless when the bug is in your understanding of shapes or the loss gradient.
  • The from-scratch version is ~60 lines — the whole architecture, readable in a coffee break.
  • It makes the 'why' of every modern layer (batchnorm, momentum, weight decay) obvious.
  • You can extend it in ways notebooks can't: custom losses, custom optimizers, mini-batch ablations.
Think about it

Mental model

Imagine

A bicycle mechanic who can strip a hub down to bearings

Think of

Implementing an MLP from scratch

Anyone can ride a bike or import nn.Module; mechanics know where each bearing sits and why grease, torque and alignment matter — which is exactly the knowledge that survives framework migration.

Mechanics

How it works

  • Init: W ~ randn ∝ 1/√fan_in, b = 0 — keeps first activations O(1).
  • Forward: z1 = X·W1+b1; h = relu(z1); out = h·W2+b2; loss = MSE or CE.
  • Backward: grad_out = (out−y)/N; dW2 = hᵀ·grad_out; dh = grad_out·W2ᵀ·(z1>0); dW1 = Xᵀ·dh.
  • Step: W ← W − lr·dW (or with momentum: keep a velocity).
  • Loop over mini-batches for epochs; print loss; hold out validation.

The full from-scratch loop in six boxes.

Init
Forward
Backward
Step
Batch loop
Sanity
Real world

In production

  • Teaching/troubleshooting in repos and interviews — the canonical 'explain neural nets' artifact.
  • Ultra-light deployments (<1KB) where a 2-layer net on tabular data beats importing a framework.
  • Simulation/augmentation in RL and on-device code where autograd can't be trusted to exist.
Guidelines

Best practices

  • Overfit a single batch before full training; if loss can't go near zero there, plumbing is broken.
  • Implement numeric gradient check once per layer for the first 200 samples.
  • Keep shapes explicit ('(N, h)') in comments beside each equation.
  • Separate pre-activation from activation (relu(z) vs h) so backward conditions read naturally.
Watch out

Common mistakes

  • Mixing relu-on-h with relu-on-z in the backward mask — wrong derivative.
  • Forgetting to divide gradient by N so loss doesn't balloon with batch size.
  • Updating weights before computing gradients for all layers (asymmetric update).
  • Ignoring the bias term in backward (bias grad = row sums of incoming gradient).
Root cause analysis

Common issues

Error
Loss diverges to a huge value immediately
Root cause

Init scale or LR too big for the data scale

Fix

Scale init with 1/√fan_in, normalize X, and drop LR by 10x.

Error
Both dW1 and dW2 are exactly zero
Root cause

Relu mask (z1>0) all False (saturated dead layer) or zero LR

Fix

Check z1 values and LR; use leaky-relu/init-scale to revive activations.

Error
Numeric grad check fails for a layer
Root cause

Backward formula mismatch (wrong mask or missing division by N)

Fix

Isolate with one layer and compare analytic vs finite-difference gradients.

When things break

Debugging tips

  • Check dW1, dW2 have the exact shapes of W1, W2.
  • Numeric-check: (f(w+ε)−f(w−ε))/(2ε) ≈ dW for a random sample — to 1e-4 or better.
  • Print loss for a fixed init; if it differs from a framework equivalent, diff layer by layer.
  • Add weight-norm prints; exploding norms mean the update or init scale is off.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build and Train an MLP in Pure NumPy
35 min · advanced
In progress
Objectives
  • Implement init, forward, backward and optimizer updates in numpy
  • Overfit a single batch to prove the plumbing works
  • Pass a finite-difference check on one layer's gradients

Build and Train an MLP in Pure NumPy

35 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What is the correct gradient of the hidden layer for a ReLU MLP?

2

You must divide the batch gradient by the number of samples so the loss scale stays independent of batch size.

Go deeper

Official documentation

karpathy/nn-zero-to-hero
video