Mathematics
Intermediate 15 min read Mathematics

Derivatives and Gradients

The learning rule behind every model

A derivative is a rate of change; the gradient generalizes it to many inputs. Gradient descent walks downhill on the loss surface, which is how nearly every ML model learns.

derivativegradientslopegradient descentloss surfacelearning rate
Prerequisites: Why Mathematics Matters
Before we start

What you will learn

  • Interpret a derivative as local slope
  • Compute the gradient of a simple loss by hand and with finite differences
  • Explain the gradient descent update w ← w − η·∇L
Introduction

What is this?

The derivative f'(x) measures how fast f changes as x changes — the slope of the tangent line. With many inputs, the gradient ∇f is a vector of all partial derivatives, pointing in the direction of steepest increase. Gradient descent moves parameters in the opposite direction — steepest decrease — repeatedly, until the loss bottoms out.

Motivation

Why do you need it?

  • Training means minimizing loss, and gradients tell you which way is downhill.
  • The learning-rate choice is entirely about gradient step sizes.
  • Vanishing gradients (product of many small derivatives) explain why deep networks are hard to train.
  • Understanding the update rule lets you diagnose slow/exploding training.
Think about it

Mental model

Imagine

Descending a foggy mountain

Think of

Gradient descent on the loss landscape

You can't see the valley, only the slope under your feet. Feel the steepest downhill (gradient), take a step proportional to it (learning rate), and repeat. Big steps risk overshooting; small steps crawl.

Mechanics

How it works

  • For a univariate f, f'(x) = limₕ→₀ [f(x+h) − f(x)] / h — ratio of output change to input change.
  • The gradient ∇L(w) stacks ∂L/∂wᵢ for every parameter — the steepest-ascent direction.
  • The update w ← w − η∇L(w) moves parameters against the gradient to reduce loss.
  • Each full pass over data computes gradients and updates; an epoch is one such pass.
  • SGD uses one random sample's gradient; mini-batch uses a subset; full-batch uses all data — a variance/computation tradeoff.

Walking the loss curve to the minimum.

Start
Compute gradient
Step downhill
Repeat
Converge
Hands-on

Examples

Numerical gradient check on a simple loss

Numerical gradient check on a simple loss
import numpy as np
def loss(w): return (w - 3) ** 2   # min at w=3
def grad_num(w, h=1e-5):
    return (loss(w + h) - loss(w - h)) / (2 * h)
 
w = 10.0
eta = 0.1
for i in range(40):
    w = w - eta * grad_num(w)
print(round(w, 4))   # 3.0066 — converged near the minimum

The numerical gradient (central difference) estimates the slope, and the update nudges w downhill. With η=0.1 it converges from 10 to ~3. Real models use analytic gradients (backprop), but finite differences are invaluable for checking them.

Real world

In production

  • A deep-learning team uses gradient clipping when 50-layer networks explode — the L2 norm of the gradient is capped before the update.
  • An MLOps runbook checks learning-rate schedules after observing loss plateaus at 0.1 but converging at 0.01.
  • An NLP pipeline measures per-layer gradient norms to find where signal dies (vanishing gradients).
Guidelines

Best practices

  • Start with small learning rates and log them in experiment trackers.
  • Use gradient clipping for recurrent or deep models.
  • Validate hand-written gradient code against finite differences once before trusting it.
Watch out

Common mistakes

  • Learning rate too large → loss diverges to NaN; too small → it crawls.
  • Updating parameters in the wrong direction (w += η∇L) — always subtract the gradient.
  • Ignoring the gradient norm so a silent scale problem delays convergence for hours.
When things break

Debugging tips

  • Plot loss vs epoch; a flat line hovering means the learning rate is too small or the gradient vanished.
  • If loss is NaN after step N, inspect whether the step was too large (lower η) before suspecting data.
  • Print gradient norms periodically — they should trend down, not blow up.
Interviews

Interview checkpoint

Practice

Hands-on lab

Watch Gradient Descent Converge
25 min · intermediate
In progress
Objectives
  • Implement gradient descent from scratch on a convex loss
  • Observe how learning rate changes convergence
  • Detect divergence vs overshoot empirically

Watch Gradient Descent Converge

25 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The gradient of the loss points:

2

You should add the gradient to subtract the loss.

Go deeper

Official documentation

3Blue1Brown — Gradient descent
video
scikit-learn SGD for linear models
docs