Neural Networks
Beginner 13 min read Neural Networks

What a Neural Network Computes

Weighted sums all the way down

A neural network is a stack of weighted ladders: inputs pass through layers of weighted sums + activations to a prediction. Understanding the forward pass is the base layer for everything from CNNs to transformers.

neural networkforward passlayersweightsactivationparameters
Prerequisites: The ML Pipeline
Before we start

What you will learn

  • Describe the forward pass through a layer
  • Explain parameters vs activations vs layers
  • Build a tiny network by hand
Introduction

What is this?

A neural network takes a vector of numbers (a document embedding, pixel values, features) and pushes it through layers. Each layer computes weights × inputs plus a bias, passes the result through a nonlinear activation, and hands the output to the next layer. The final layer produces the prediction. Every modern model — including transformers — is this same mechanism arranged at scale.

Motivation

Why do you need it?

  • LLMs are neural networks; you cannot reason about capacity, cost or errors without the forward-pass mental model.
  • Attention, embeddings and logits are all linear maps + nonlinearities with bigger bookkeeping.
  • Understanding layer widths helps reason about compute and memory footprints.
  • Debug surprises (NaN, saturation) trace back to layer math.
Think about it

Mental model

Imagine

An assembly line of weighted votes

Think of

A neural network layer

Each station takes every part from the previous station, weighs each one's importance, sums the votes, applies an on/off-ish decision, and passes its verdict on. Stack stations and you build predictions from parts.

Mechanics

How it works

  • Inputs x form a vector; each layer has a weight matrix W and bias b.
  • The layer computes z = W·x + b — a weighted vote across all inputs.
  • A nonlinearity (ReLU, tanh) transforms z; without it the whole stack collapses to one linear map.
  • Stacked layers compose into richer features: edges → shapes → objects (vision), tokens → phrases → meaning (text).
  • The final layer maps to logits; softmax/excitation turns logits into probabilities.

The forward pass.

Inputs
Layer 1
Layer 2
Output layer
Prediction
Hands-on

Examples

Two scaled conjuncts by hand

Two scaled conjuncts by hand
import numpy as np
 
x = np.array([0.5, 0.3])            # input features
W1 = np.array([[0.2, -0.4], [0.1, 0.3]])
b1 = np.array([0.0, 0.1])
z1 = W1 @ x + b1
a1 = np.maximum(0.0, z1)           # ReLU
W2 = np.array([[1.0, -1.0]])
out = W2 @ a1
 
print(out)  # a real-valued head output from a 2-layer net

Exactly one forward pass end to end. Scale the weights and stack thousands of layers, and you have the compute a transformer performs per token.

Real world

In production

  • An embeddings model is a network whose hidden activations become the vectors you retrieve against.
  • A classifier head trained on a frozen embedding is one linear layer over a pre-trained base.
  • Serving-size math (how many params, what precision) derives directly from layer widths.
Guidelines

Best practices

  • Reason in layers and parameter counts, not 'model size' vaguely.
  • Profile the forward pass for latency before optimizing anything else.
  • Use standard activations unless you have measured reason not to.
Watch out

Common mistakes

  • Believing 'deep' always beats 'wide' without measuring the task.
  • Confusing activations (per-input) with parameters (static weights).
  • Hand-rolling forward passes in app code instead of using the library's optimized path.
Root cause analysis

Common issues

Error
All outputs identical regardless of input
Root cause

Activation saturation or loss of gradient signal

Fix

Check for scale issues, dead ReLUs and normalization.

When things break

Debugging tips

  • NaN in predictions usually means activations exploded — check normalization and learning rate, not the data.
  • A frozen feature layer + fine head regressing? Verify the gradient actually flows (dummy-training a single batch).
Interviews

Interview checkpoint

Practice

Hands-on lab

Forward Pass in Pure Python
30 min · intermediate
In progress
Objectives
  • Compute a 2-layer network forward pass with no libraries
  • Verify shapes and output sanity by hand
  • Trace one weight's influence on the output

Forward Pass in Pure Python

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which quantity is adjusted during training?

2

A stack of linear-only layers can model any nonlinear function.

Go deeper

Official documentation

Deep learning course (3Blue1Brown)
video
PyTorch neural network tutorial
docs