Mathematics
Intermediate 15 min read Mathematics

Partial Derivatives and Gradient Descent

Optimizing functions of many variables

Partial derivatives measure how a multi-input function changes along one input at a time; stacking them produces the gradient that drives gradient descent for real models.

partial derivativegradientchain rulebackpropagationmultivariate calculus
Prerequisites: Derivatives and Gradients
Before we start

What you will learn

  • Compute partial derivatives of typical loss functions
  • Assemble the gradient and apply a multivariate update
  • Explain backpropagation as the chain rule over a model's computation graph
Introduction

What is this?

A function of several variables has many slopes — one per input. The partial derivative ∂L/∂wᵢ treats every other input as constant and differentiates with respect to wᵢ. The gradient is the vector of all partials. Real models have millions of parameters, so computing all partials efficiently is the role of backpropagation, which is simply the chain rule applied to a computation graph.

Motivation

Why do you need it?

  • Every sklearn/sklearn-compatible model minimizes a multivariate loss with partial derivatives.
  • Backpropagation exists because naive numerical differentiation is far too slow at millions of parameters.
  • Understanding partials demystifies per-feature sensitivity: which inputs move the loss most.
  • It's the same math whether the 'model' is linear regression or a 1-billion-parameter transformer.
Think about it

Mental model

Imagine

A hiker on a rolling hill

Think of

Partial derivatives on a multivariable loss

Slope north is ∂h/∂north, slope east is ∂h/∂east. Hiking straight downhill needs the combination — the gradient vector. Each partial is just one direction's slope.

Mechanics

How it works

  • ∂L/∂wᵢ treats all other variables as constants and differentiates normally.
  • The gradient ∇L = [∂L/∂w₁, ..., ∂L/∂wₙ] is the steepest-ascent vector at the current point.
  • Update: w ← w − η∇L, applied element-wise to every parameter.
  • Backprop computes gradients by walking the computation graph backward, multiplying derivatives with the chain rule (the product rule of errors).
  • Autodiff libraries (PyTorch, JAX, TensorFlow) do exactly this bookkeeping for you — behind a .backward() call.

Gradient of a two-parameter loss: L(w₁, w₂).

Loss L(w₁,w₂)
∂L/∂w₁
∂L/∂w₂
Gradient vector
Update
Hands-on

Examples

Closed-form partials for mean squared error

Closed-form partials for mean squared error
import numpy as np
X = np.array([[1.0],[2.0],[3.0]])
y = np.array([2.0, 4.0, 6.0])          # y = 2x
w, b = 0.0, 0.0                        # start at 0
 
for _ in range(200):
    y_hat = X @ w + b
    err = y_hat - y
    dw = (2 / len(y)) * (X.T @ err)    # partial wrt w
    db = (2 / len(y)) * err.sum()      # partial wrt b
    w -= 0.05 * dw; b -= 0.05 * db
print(w.item(), round(b, 4))           # ~2.0, ~0.0

Each partial derivative is derived once on paper, then applied as a vectorized update: dw is the gradient of MSE with respect to w, db with respect to b. This is linear regression trained by gradient descent.

Real world

In production

  • Deep networks train via backprop, the exact chain-rule partials computed backward through layers.
  • Gradient clipping in production transformers caps the norm of the assembled gradient vector.
  • Feature-importance tools (SHAP) approximate per-feature partial sensitivity at each prediction.
Guidelines

Best practices

  • Derive partials for a toy loss once by hand to anchor the intuition.
  • Use autodiff frameworks for anything beyond toy models — hand-coded gradients drift.
  • Verify gradients with a numerical check before pushing an experiment pipeline.
Watch out

Common mistakes

  • Forgetting that the partial for the bias has a different form than the partial for the weight.
  • Summing gradients across the batch without dividing by batch size (makes the step scale with batch size).
  • Trusting a hand-written gradient silently — one sign error can make loss climb.
When things break

Debugging tips

  • Compare analytic gradient vs finite differences on one mini-batch; relative error < 1e-5 is healthy.
  • If training diverges only at large batch sizes, check whether you normalized by batch size.
  • Log per-parameter gradient norms to see which layer/parameter receives no signal.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

For L(w₁,w₂) = w₁² + w₂², what is the gradient at (3, 4)?

2

Backpropagation requires symbolic differentiation written by hand.

Go deeper

Official documentation

Backpropagation (3Blue1Brown)
video
Automatic differentiation — JAX
docs