Neural Networks
Intermediate 12 min read Neural Networks

Loss Functions

Defining 'wrong' precisely

The loss function quantifies wrongness and steers every gradient. Classification uses cross-entropy; regression uses MSE/MAE; generation uses next-token cross-entropy. Choosing right is choosing what 'better' means.

losscross-entropyMSEMAEnext-token predictionobjective
Prerequisites: Backpropagation & Learning
Before we start

What you will learn

  • Match loss functions to task types
  • Explain why cross-entropy underpins LLMs
  • Diagnose loss pathologies that waste training
Introduction

What is this?

A loss function scores how wrong each prediction is — the model gets smaller values when closer to right. The gradient of the loss is literally the training direction. Classification usually uses cross-entropy; regression typically MSE or MAE; and language models train on next-token cross-entropy, which is why 'LLM' and 'estimating text probability' are the same idea.

Motivation

Why do you need it?

  • The loss defines what the model considers good — wrong loss, wrong model.
  • Cross-entropy works perfectly with softmax logits and labels.
  • MSE punishes outliers quadratically; MAE is robust. The choice changes behavior.
  • Every generation quality effort starts from the next-token cross-entropy objective.
Think about it

Mental model

Imagine

A referee with a strict scorebook

Think of

The loss function

The referee decides exactly how many points an off-target move loses. The fun discipline of backprop just reads those points and runs downhill — the whole competition is shaped by the scoring rubric.

Mechanics

How it works

  • Regression: MSE = mean (y−ŷ)², sensitive to outliers; MAE = mean |y−ŷ|, robust.
  • Classification: cross-entropy = −Σ y log(p), perfect fit with softmax probabilities.
  • LLM pre-training: predicts each next token with cross-entropy over the vocabulary.
  • Hinge/ranking losses favor correct ordering over magnitude.
  • Weighted losses reshape priorities toward rare or expensive mistakes.

Task → head → loss.

Regression
Binary class
Multiclass
Language
Real world

In production

  • Fraud classifiers weight the rare-class loss so misses cost more than flags.
  • Object detection masks use GIoU-family losses to optimize geometry, not pixels.
  • LLM fine-tuning matters most through data/objective structure — same CE loss, better target distribution.
Guidelines

Best practices

  • Match the loss to the decision, not to vibes: robust vs single-error dominated.
  • Weight class importance inside the loss explicitly.
  • Before fancy losses, sanity-check that your CE sees the right label distribution.
Watch out

Common mistakes

  • Using MSE on a classification and waiting for 'convergence'.
  • Ignoring NaN/Inf when logits are ill-posed (e.g. tiny negatives in logs).
  • Believing 'custom loss' fixes data problems.
Root cause analysis

Common issues

Error
NaN cross-entropy for a softmax head
Root cause

Logits inflated or labels out of range

Fix

Stabilize (log-softmax), clamp, and verify label index bounds.

Error
MSE dominated by a few points
Root cause

Heavy-tailed targets

Fix

Use Huber/MAE or winsorize extreme targets.

Error
Model optimizes loss but product metric doesn't move
Root cause

Loss proxy diverges from the business metric

Fix

Add an auxiliary loss or re-frame the objective around the metric.

When things break

Debugging tips

  • Loss plateaus but big for softmax — usually logit scale or label mismatch, not 'still learning'.
  • MSE sensitive to one huge error — plot residuals to find the outlier.
  • If CE is NaN, check label classes exist in the vocabulary and logits aren't all −Inf.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which loss naturally pairs with softmax classification?

2

MAE is more robust to outliers than MSE.

Go deeper

Official documentation

PyTorch loss functions
docs