Neural Networks
Intermediate 15 min read Neural Networks

Multi-Layer Perceptron (MLP)

Depth turns one boundary into infinitely many curves

An MLP stacks layers of neurons with nonlinear activations, allowing curved decision boundaries. With one hidden layer it already universal-approximates; adding depth trades efficiency for expressiveness.

MLPhidden layerfeedforwardneural networkactivationuniversal approximation
Prerequisites: The Perceptron Deep Learning Basics
Before we start

What you will learn

  • Describe the anatomy of an MLP and how layers combine
  • Explain why one hidden layer already gives universal approximation
  • Estimate parameter counts and capacity trade-offs
Introduction

What is this?

A Multi-Layer Perceptron is several perceptron-style layers chained (input → hidden → hidden → output), each with a nonlinear activation between linear transforms. Because each nonlinear layer re-shapes the input space, the final layer sees a transformed representation where even XOR is linearly separable. One hidden layer can approximate any continuous function; depth primarily makes it efficient and learnable to fit.

Motivation

Why do you need it?

  • MLPs solve the perceptron's XOR blind spot by adding only one middle layer.
  • They're the backbone of tabular deep models and the head on top of CNNs/transformers.
  • Parameter math is simple — you can predict capacity before training.
  • Every deeper model your career touches has an MLP at its output stage.
Think about it

Mental model

Imagine

A team of translators, each relaying in a new language

Think of

An MLP's hidden layers

The input arrives in raw coordinates; the first translator rewrites it, the second rewrites again, and by the output layer the meaning (class) has been re-expressed in a form the final judge can easily read off.

Mechanics

How it works

  • Layer 1: z1 = X·W1 + b1, then h1 = activation(z1), producing internal features.
  • Layer 2 (and more): repeat; hidden layers re-map the input toward separability.
  • Output layer: one linear map per class (logits), then softmax/sigmoid to probabilities.
  • Parameters: (in·h1 + h1) + (h1·h2 + h2) + ... — width grows the count fastest.
  • Training via backprop adjusts all weights to minimize the final loss end-to-end.

An MLP with one hidden layer.

Input x
Hidden units
Output units
Probabilities
Prediction
Real world

In production

  • Tabular classifiers where gradient models (GBDT) fail: multi-class heads, embeddings, mixed features.
  • Any classifier/regressor as the readout head on CNN/transformer embeddings.
  • Small serveable models on edge/MCU-class devices with 1-2 hidden layers.
Guidelines

Best practices

  • Normalize inputs (standard scaling) and use reasonable init within each layer.
  • Start with one hidden layer at moderate width; add depth/width only when validation demands it.
  • Use softmax/CE for classification, identity/MSE for regression, matching activation to loss.
  • Regularize (weight decay, dropout) once width exceeds data size.
Watch out

Common mistakes

  • Making hidden layers deeper than data warrants — an overparameterized curve-fit on small tables.
  • No normalization: MLPs are notorious for punishing unscaled, huge-magnitude features.
  • Treating width as the only dial and ignoring LR/regularization — capacity isn't free.
  • Reading validation accuracy as proof of generalization when the test set was reused for tuning.
Root cause analysis

Common issues

Error
MLP does no better than logistic regression despite hours of tuning
Root cause

Data too small or too noisy, or features not normalized

Fix

Normalize/scale features, add regularization; be honest that shallow models may own the tabular baseline.

Error
Sigmoid-heavy MLP gets stuck around 0.5 confidence
Root cause

Saturated hidden activations from deep chain + no normalization

Fix

Switch to ReLU hidden layers with proper init and batchnorm/layernorm.

Error
Validation accuracy spikes and crashes between runs
Root cause

Uncontrolled seed/shuffle with small data

Fix

Fix seeds, cross-validate, and average results over a few random restarts.

When things break

Debugging tips

  • Check the first layer's weight/activation scale after a few steps using a printout — normalization holds clues.
  • Overfit one batch to double-check plumbing, then introduce the full set.
  • Compare against the single-perceptron baseline; if no better, the MLP is overfitting or starved for data.
  • Count parameters vs samples; if params >> rows, expect training-time memorization on the val curve.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Where does an MLP's power over a single perceptron come from?

2

An MLP with one hidden layer can approximate any continuous function with enough units.

Go deeper

Official documentation

Neural Networks — sklearn docs
docs
Universal approximation theorem
docs