Intermediate 13 min read Regression

Gradient Descent

The hill-walking optimizer

Gradient descent walks parameter space downhill along the loss gradient — the engine behind large linear models, neural nets and most of modern ML at scale.

gradient descentlearning ratestochasticSGDloss minimizationconvergence

Prerequisites: Linear Regression

Before we start

What you will learn

Explain the update rule w := w - η ∇loss(w)
Diagnose learning-rate failure modes: divergence, slow crawl, oscillation
Distinguish batch, mini-batch and stochastic gradient descent

Introduction

What is this?

Gradient descent minimizes a loss by repeatedly stepping parameters in the direction that most reduces it: w ← w - η∇L(w). Batch GD uses the whole dataset per step; mini-batch (SGD) uses random mini-batches — cheaper, noisier, faster to scale. Learning rate η is the decisive knob: too large diverges, too small crawls.

Motivation

Why do you need it?

Closed-form OLS fails beyond a size; GD scales to millions of rows and arbitrary models.
Every neural network, GBM gradient step, embedding, and autoencoder is GD under the hood.
Understanding its failure modes explains most 'loss going to NaN' mysteries.
Adaptive variants (Adam, RMSProp) are just GD with per-parameter step shaping.

Think about it

Mental model

Imagine

Descending a foggy mountain by feel

Think of

Gradient descent steps

Feel the slope underfoot (gradient), step downhill (update), and re-feel. Too big a step leaps off a cliff (diverge); too small a step takes all week (crawl); rolling down into a bowl you settle at the local best.

Mechanics

How it works

Define loss L(w) (MSE for regression, cross-entropy for classification).
Compute gradient ∇L = derivative of the loss w.r.t. each parameter.
Update w := w - η ∇L; repeat until the gradient is near zero.
Batch: full-data gradient (precise, slow). Mini-batch: small random samples (fast, noisy — often good!). Stochastic: single rows.
Tune η (try 0.01-0.001 scales), monitor loss per epoch for divergence or plateau.

Learning-rate behavior on a bowl.

Too largeloss oscillates/explodes

Too smallslow crawl to minimum

Just rightsteady descent

Plateaugradient near zero iterations

Hands-on

Examples

Manual GD vs library fit on a simple slope

import numpy as np
 
rng = np.random.RandomState(0)
x = rng.uniform(0, 5, 300)
y = 2.0 * x + 1.0 + rng.normal(0, 0.5, 300)
 
w, b, lr = 0.0, 0.0, 0.1
for epoch in range(50):
    pred = w * x + b
    dw = 2 * np.mean((pred - y) * x)
    db = 2 * np.mean(pred - y)
    w -= lr * dw
    b -= lr * db
    if epoch % 10 == 0:
        loss = np.mean((pred - y) ** 2)
        print(f"epoch {epoch:>2}: w {w:.3f} b {b:.3f} loss {loss:.3f}")
print("learned w,b:", round(w, 3), round(b, 3), "(true 2.0, 1.0)")

Fifty gradient steps drag (w,b) toward the true (2, 1) — the same update every framework runs at scale, just hand-rolled.

Real world

In production

Training large linear models where OLS matrix inversions become intractable.
Neural networks: adaptive SGD (Adam) is the default engine.
Embeddings and matrix factorization: SGD batches stream user-item pairs efficiently.

Guidelines

Best practices

Scale features so gradients behave; normalize the learning rate per layer where model-specific; monitor the loss curve every epoch — a loss that plateaus early may need schedule decay; fix the random seed for reproducible runs.

Watch out

Common mistakes

Massive learning rate causing NaN (start smaller / clip); no feature scaling so gradients vary by feature; evaluating the loss after each mini-batch instead of the epoch; stopping too early on a noisy curve.

When things break

Debugging tips

Loss to NaN: reduce lr 10x, check for inf gradients (overflow), clip.
Slow convergence: raise lr, normalize inputs, or use momentum/Adam.
Bouncing loss: mini-batch noise — this can be fine; look at the epoch-averaged curve.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

The GD update rule is:

Loss explodes to NaN mid-training. First suspect?

Go deeper

Official documentation

SGDRegressor (sklearn)

docs

Stochastic gradient descent | Wikipedia