CNN
Intermediate 15 min read CNN

Transfer Learning

Standing on the shoulders of a pretrained giant

Transfer learning reuses a pretrained model's learned features for a new task: freeze the spine, replace the head, fine-tune. It converts 'need a million images' into 'need a few thousand'.

transfer learningfine-tuningfeature extractionfrozen layerspretraineddomain adaptation
Prerequisites: CNN Architecture Modern Architectures
Before we start

What you will learn

  • Explain what pretrained features buy you and how they transfer
  • Choose between frozen-extraction and full fine-tuning
  • Avoid the classic transfer pitfalls (forgetting, catastrophic forgetting, domain gap)
Introduction

What is this?

Transfer learning starts from a model pretrained on a huge, general dataset (ImageNet) and adapts it to your task. The shallow conv layers learn generic edges/textures that transfer broadly; only the last head is task-specific. You either freeze the spine and train just the head (feature extraction) or unfreeze and fine-tune everything with a small LR, using pretrained weights as initialization.

Motivation

Why do you need it?

  • Training a CNN from scratch on 5,000 images overfits; pretrained features generalize.
  • Transfer converges faster — your data only adjusts the tail of the feature hierarchy.
  • It's cheap: a small head trains in minutes; full fine-tune trains in a few GPU-hours.
  • It creates a practical baseline that archival-scratch training almost never beats at small scale.
Think about it

Mental model

Imagine

A trained chef learning a regional menu

Think of

Transfer learning

The chef already knows knife work, heat control and how ingredients behave (the pretrained layers). Learning the new menu is mostly new recipes (the head), and a light retraining of a few signature dishes (fine-tuning) — not restarting from 'boil water'.

Mechanics

How it works

  • Load a pretrained model; drop the final classifier; attach your head (matching class count).
  • Freeze the backward weights (requires_grad=False) and train only the head — feature extraction.
  • Or unfreeze with a smaller LR for fine-tuning — adapts mid-level features to your domain.
  • Stage-based FT: train head first, then unfreeze deeper layers with a lower LR (e.g. 1/10 the head's).
  • Keep the same input preprocessing used in pretraining (normalize, resize, channel order).

Pretrained spine + new head.

ImageNet spine
New head
Head-only first
Unfreeze deeper
Validate
Real world

In production

  • Medical, retail, satellite and defect-detection vision almost always start from a pretrained backbone.
  • Product categorization pipelines fine-tune ResNet/EfficientNet heads per client.
  • Same principle powers NLP/HF: BERT/CLIP embeddings replace hand features everywhere.
Guidelines

Best practices

  • Match preprocessing to the pretrained source exactly (mean/std, resize, BGR/RGB).
  • Always train the head first on frozen features; unfreeze only if validation demands it.
  • Unfreeze late layers with a 10x-smaller LR than head training.
  • Use label smoothing/dropout in the head to combat small-data overfit.
Watch out

Common mistakes

  • Forgetting imagenet preprocessing — a huge, silent accuracy hit.
  • Fine-tuning the whole model from step 0, trashing the pretrained features early.
  • Keeping the pretrained classifier head and only changing inputs — class count and semantics mismatch.
  • Judging transfer 'failed' before doing the head-first warm-up stage.
Root cause analysis

Common issues

Error
Transfer model performs like random guessing
Root cause

Input preprocessing mismatch (normalization, resize, channel order)

Fix

Re-run your exact import+preprocess against the reference upstream (torchvision transforms).

Error
Fine-tuning destroys pretrained knowledge
Root cause

LR too high on unfrozen layers (catastrophic forgetting)

Fix

Lower the fine-tune LR by 10-100x and unfreeze gradually.

Error
Accuracy plateaus right after head training
Root cause

Domain features mid-levels not adapted enough

Fix

Unfreeze last 1-2 stages with low LR; consider intermediate feature extraction.

When things break

Debugging tips

  • Sanity: evaluate the frozen spine's head on 1 batch of your train — should be well above chance (else preprocessing is wrong).
  • Compare feature-extraction vs full fine-tune on the same val split to attribute gains.
  • Print which stages are frozen in config to avoid 'supposedly unfrozen' surprises.
  • Watch val loss in the first fine-tune epochs: a sharp drop means features adapted well.
Interviews

Interview checkpoint

Practice

Hands-on lab

Fine-Tune a Pretrained Model
35 min · intermediate
In progress
Objectives
  • Load a pretrained torchvision model with a custom head
  • Freeze the spine and train the head on a small dataset
  • Unfreeze partway and compare validation curves

Fine-Tune a Pretrained Model

35 min · intermediate

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What does freezing the backbone do?

2

Transfer learning requires re-specifying the model's input preprocessing to match the pretrained source.

Go deeper

Official documentation

PyTorch transfer learning tutorial
docs