Deep Learning
Intermediate 16 min read Deep Learning

Loss Functions & Optimizers

What you minimize and how

The loss scores every prediction and the optimizer decides how to descend it. Choosing them well is the difference between a loss curve that falls smoothly and one that oscillates, plateaus or diverges.

loss functionoptimizercross entropyMSEAdamSGDlearning rateconvergence
Prerequisites: Backpropagation Activation Functions
Before we start

What you will learn

  • Match loss functions to problem type and output activation
  • Explain how SGD, momentum and Adam differ in practice
  • Tune learning rate using convergence signals instead of guesses
Introduction

What is this?

The loss quantifies how wrong the model is on a batch, and the optimizer uses its gradient to update weights. Loss choice must match the problem (classification → cross-entropy, regression → MSE/MAE) and the final activation. Optimizers build from plain SGD by adding momentum and per-parameter adaptive rates (Adam), changing step size over training. They are inseparable: the best loss trained with a bad optimizer still won't converge.

Motivation

Why do you need it?

  • Wrong loss/activation pair silently cripples gradients (e.g. MSE on sigmoids).
  • Momentum smooths noisy gradients; Adam adapts learning rates per parameter for sparse, varied scales.
  • Learning rate is the single most impactful knob — too high diverges, too low stalls.
  • A optimizer/loss audit is the fastest fix for 'loss is flat' complaints.
Think about it

Mental model

Imagine

A hiker descending a foggy mountain

Think of

An optimizer minimizing the loss landscape

The hiker needs a direction (gradient), step size (learning rate), and rhythm (momentum) — and in fog, particular features need different step sizes (Adam's per-parameter rates). Step too big and you overshoot; too small and night falls before you reach the valley.

Mechanics

How it works

  • A loss scores a batch: CE = -Σ y·log(p), MSE = mean((ŷ-y)²); both are differentiable so gradients exist.
  • SGD: w ← w - lr·g, using the current batch's gradient (noisier, but cheap and generalizes).
  • Momentum: keeps a velocity v = β·v + g and steps with v — dampens noise, accelerates saddles.
  • Adam: per-parameter adaptive rates from first and second moment estimates, with bias correction — robust, fast early.
  • Schedule: lower the LR over time (step/decay/cosine) so coarse descent becomes fine-tuning near the optimum.

From loss to optimizer to weights and back.

Predictions
Loss
Gradient
Optimizer
Repeat
Real world

In production

  • Classification heads couple CrossEntropy/BCEWithLogits with the right final layer — standard across image and NLP models.
  • Regression uses MSE for L2 or Huber for robustness to outliers; quantile losses for intervals.
  • Recommendation & retrieval train with sampled softmax / InfoNCE — pairwise or contrastive losses.
  • LLM pretraining/fine-tuning uses cross-entropy over token logits with AdamW.
Guidelines

Best practices

  • Pair loss with activation (logits + CE; identity + MSE) to keep gradients well-conditioned.
  • Default to Adam/AdamW for new work, SGD+momentum for image nets or when generalization is finicky.
  • Start with a small LR and double until divergence to find the effective-max LR (LR range test).
  • Decay LR on plateau or use cosine schedule; never hold a constant high LR to the end.
Watch out

Common mistakes

  • Sticking with MSE on a classification problem — gradients get tiny for wrong-but-confident predictions.
  • Tuning the architecture while leaving the optimizer at defaults, then blaming the model.
  • Using Adam without weight decay where L2-regularization-style decay is meant for SGD.
  • Changing LR too late: spikes wasted early, then tiny steps creep forever.
Root cause analysis

Common issues

Error
Train loss bounces hard every batch and never descends
Root cause

LR too high — updates overshoot the valley

Fix

Reduce LR (10x), then ramp; add gradient clipping for safety.

Error
Loss sits at exactly ln(2) for binary classification
Root cause

Confident wrong predictions with near-zero gradients, or logits↔label mismatch

Fix

Use BCE logits directly, check label range (0/1), and normalize outputs.

Error
Adam converges fast then wobbles near the end
Root cause

Adaptive LR too coarse late in training

Fix

Lower final LR, add warm-up + cosine decay, or anneal to SGD/momentum at the end.

When things break

Debugging tips

  • Watch training loss per-batch: healthy = noisy downward; diverging = ascending or NaN.
  • A flat 0.693 for binary CE means pure chance — check the activation/loss pairing first.
  • Compare SGD(with momentum) vs Adam on a small overfit run — if Adam fails a single batch, the loss gradient is wrong.
  • Log lr per epoch and weight-norm along with loss to separate schedule problems from loss problems.
Interviews

Interview checkpoint

Practice

Hands-on lab

Tune Loss and Optimizer on a Toy Problem
30 min · intermediate
In progress
Objectives
  • Train a small net with different loss/optimizer pairs
  • Observe SGD vs Adam convergence differences
  • Use the LR range test to pick a compatible learning rate

Tune Loss and Optimizer on a Toy Problem

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which loss/activation combination keeps classification gradients informative?

2

Adam removes the need to ever think about the learning rate.

Go deeper

Official documentation

PyTorch losses
docs
Adam optimizer paper
docs