CNN
Advanced 15 min read CNN

Image Segmentation

Pixel-wise understanding

Segmentation classifies every pixel: semantic assigns each pixel a class, instance separates individual objects. U-Net-style encoder-decoder or FCN heads over backbones are the workhorses.

segmentationsemantic segmentationinstance segmentationU-Netpixel-wisemask
Prerequisites: CNN Architecture
Before we start

What you will learn

  • Distinguish semantic, instance and panoptic segmentation
  • Explain the encoder-decoder = segmentation architecture
  • Pick the right loss (pixel CE vs Dice) for your task
Introduction

What is this?

Segmentation predicts per-pixel labels, producing masks instead of boxes. Semantic segmentation labels each pixel with a class (all roads 'road', all hands 'hand'); instance segmentation separates individual objects (hand #1 vs #2) with masks; panoptic combines both. Encoders (CNN/transformer) compress context, decoders upsample back to full resolution with Skip paths (U-Net) to retain fine detail.

Motivation

Why do you need it?

  • Fine-grained spatial decisions (surgical boundaries, drivable area) need pixel truth, not boxes.
  • Mask outputs power areas from precision, autonomous driving, satellite crop mapping, and medical anatomy delineation.
  • An encoder-decoder sharing one backbone extends any classification model to dense prediction.
  • It's the recognition task where evaluation (mAoU, Dice) matches the deployment metric most directly.
Think about it

Mental model

Imagine

Coloring a photo by regions of meaning

Think of

Semantic segmentation

An adult colors 'sky', 'sand', 'ocean' each with one color (semantic). Coloring the same photo once per person (this swimmer vs that swimmer) is instance segmentation. Panoptic does both maps at once.

Mechanics

How it works

  • Encoder (backbone) downsamples to a feature hierarchy, keeping context.
  • Decoder upsamples back to input resolution with skip connections from encoder stages (preserves edges).
  • Head: per-pixel logits over classes → softmax gives a class probability map.
  • Instance segmentation adds a mask branch (detect instances, then predict a mask per box).
  • Loss: pixel-wise cross-entropy, or Dice/IoU-style — Dice handles class imbalance better.

Encoder-decoder with skip connections (U-Net structure).

Input
Encoder
Bottleneck
Decoder
Output mask
Real world

In production

  • Medical: precise organ/tumor region masks (U-Net spines) analyzed by radiology workflows.
  • Autonomous vehicles: drivable-area and object-boundary masks over camera feeds.
  • Earth observation: crop type / flooding area masks from satellite imagery.
Guidelines

Best practices

  • Freeze/extend a pretrained encoder (ResNet/ViT) and add a U-Net-style decoder.
  • Resample masks with the same interpolation/precision as labels (nearest for int labels).
  • Use Dice/Jaccard loss or Dice+CE blend when classes are imbalanced (tiny tumors).
  • Evaluate on your task's metric (Dice, mIoU) and correlate with pixel accuracy — class-imbalance can fool both.
Watch out

Common mistakes

  • Using pixel accuracy as the headline metric while tiny rare classes are basically ignored.
  • Resizing/distorting masks separately from images — label/map misalignment.
  • Ignoring sparse-boundary thickness: pooling destroys thin structures without skip detail.
  • Treating a detection problem (boxes) as segmentation because it 'visually looks' like one.
Root cause analysis

Common issues

Error
Borders systematically shrunk (thin structures missing)
Root cause

Pooling/stride destroyed high-frequency detail before the decode

Fix

Add skip connections from early encoder stages and don't pool too aggressively early.

Error
Regions get matched to the wrong class
Root cause

Label-encoding mismatch or class weights missing for rare classes

Fix

Re-map class ids, add weighting/Dice and inspect a confusion matrix per class.

Error
Empty/fragmentary masks
Root cause

Weak loss on sparse structures — CE-dominated gradients

Fix

Switch to Dice or Dice+CE and add boundary-aware terms.

When things break

Debugging tips

  • Overlay predicted masks on inputs to spot off-by-one class shifts, class bleed and border shrinkage.
  • Compute per-class Dice — top-class accuracy can hide a dead rare class.
  • Check the decoder's first upsampling shape; the skips must match encoder stage spatial sizes.
  • Run the network on ONE image and iterate on that until masks are visually sane.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which task separates individual objects as distinct masks?

2

Pixel accuracy is a reliable metric for heavily imbalanced segmentation.

Go deeper

Official documentation

U-Net paper
docs
torchvision segmentation models
docs