CNN
Intermediate 15 min read CNN

Modern CNN Architectures

From LeNet to ResNet and beyond

ResNets added skip connections that let depth go from 20 to 1000 layers, and that idea underpins nearly every modern architecture — CNNs and transformers alike.

ResNetskip connectionbottleneckVGGInceptiondepthwise separableEfficientNet
Prerequisites: CNN Architecture
Before we start

What you will learn

  • Explain why skip connections fix deep-net training
  • Describe the bottleneck block and its efficiency win
  • Compare key families: VGG, ResNet, Inception/GoogLeNet, EfficientNet
Introduction

What is this?

Modern CNN design follows a wrapper on one core idea: identity skip connections. ResNet proved that adding x + F(x) lets gradients flow unimpeded and deeper stacks train; EfficientNet scaled width, depth and resolution together; depthwise separable convs cut FLOPs by an order of magnitude. Most of what you use pretrained is one of these blocks compiled neatly.

Motivation

Why do you need it?

  • Skip connections let you train realistically deep networks (30-1000+ layers) without vanishing gradients.
  • Pretrained modern spines transfer to almost every vision task with far less data and compute.
  • Architecture choice is mostly a compute/accuracy budget; knowing the trade-offs beats re-inventing.
  • The same residual idea came back with ViTs — it's the deep-vision workhorse.
Think about it

Mental model

Imagine

A detour-free highway for error reports

Think of

Residual skip connections

Without skips, an error report (gradient) crawls back through every narrow side street (layer products) and arrives exhausted. ResNet adds a wide highway around the block so the report, and the learning signal, arrive intact even from 100 layers away.

Mechanics

How it works

  • Residual block: out = x + conv(conv(x)); the additive identity lets the block learn just the residual F(x)≈0 initially.
  • Bottleneck: instead of 3×3→3×3, use 1×1 (compress) → 3×3 (convolve) → 1×1 (expand), cutting FLOPs.
  • VGG: uniform 3×3 stacks — simplicity first, decent accuracy at high compute.
  • Inception: parallel 1×1/3×3/5×5/pool branches merged — variety of receptive fields per block.
  • EfficientNet: NAS-found scaling of depth×width×resolution in one compound factor.

Residual block vs plain block.

Input x
Conv-ReLU-Conv
Add x
Identity path
Output
Real world

In production

  • ResNet-50/152 and EfficientNet are default pretrained backbones in torchvision and TensorFlow Hub.
  • Detection/segmentation backbones (Faster R-CNN, DeepLab) reuse ResNet stages.
  • MobileNet/depthwise blocks power on-device models under a few megabytes.
Guidelines

Best practices

  • Use pretrained weights for any task with less than a few hundred thousand images.
  • Start with ResNet-50-class capacity; scale up only after you've rule out data/optimization issues.
  • Prefer depthwise-separable (MobileNet/EfficientNet-Lite) when serving at the edge.
  • Freeze the early stages and fine-tune later stages for fast domain adaptation.
Watch out

Common mistakes

  • Over-thinking architecture on a small dataset when a pretrained ResNet + fine-tune wins 95% of the time.
  • Ignoring input resolution when porting an architecture (224 vs 512 changes FLOPs quadratically).
  • Removing residual connections 'for simplicity' then struggling to train.
  • Using a huge backbone for a tiny problem, paying inference seconds for no gain.
Root cause analysis

Common issues

Error
Shape error on the residual add
Root cause

Stride/skip channel mismatch (x has different spatial or channel size than F(x))

Fix

Add a strided 1×1 conv to project the skip to the block's output size.

Error
Fine-tuned backbone barely changes weights
Root cause

Batchnorm running stats not updated (train flag off in 'training' path)

Fix

model.train() and select batchnorm momentum properly for transfer.

Error
Better architecture is 2x slower with no accuracy gain
Root cause

Bottleneck vs plain trade-off misapplied at your resolution

Fix

Benchmark FLOPs and pick the family that fits the compute budget.

When things break

Debugging tips

  • Verify the skip dimension matches (stride changes need 1×1 conv on the skip).
  • Inspect grad-norm in the last conv of a block vs the skip path — skips should carry most early.
  • When porting architecture, replicate the exact normalization placement before tuning.
  • Check that global-pooled vector length matches your head after removing pretrained head.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

The core innovation of ResNet is:

2

A 1×1 convolution can only change dimensions, never mix channel information.

Go deeper

Official documentation

ResNet paper
docs
EfficientNet paper
docs