CNN
Advanced 15 min read CNN

CNNs in Production

From notebook accuracy to serving reliability

Serving CNNs adds a production-layer of concerns: export/quantization, batch inference, latency budgets, input pipelines, and how accuracy holds up at real-world resolutions and edge hardware.

servingquantizationONNXlatencybatch inferenceedge inferencemodel export
Prerequisites: CNN Architecture Transfer Learning
Before we start

What you will learn

  • Choose an export/serving path (TorchScript, ONNX, TensorRT, TFLite)
  • Understand quantization's latency/perf trade-offs
  • Design inference pipelines that meet latency and throughput budgets
Introduction

What is this?

Production CNN serving is about turning a trained model into a fast, reliable, deployable artifact. This means exporting to an optimized runtime (ONNX/TensorRT/TFLite), choosing between full precision and quantized weights, batching requests to keep GPUs busy, and checking the metrics that matter (p50/p95 latency, throughput, drift in deployed input). Image preprocessing at serve time is just as load-bearing as training-time augmentation.

Motivation

Why do you need it?

  • Flops grown quadratically with resolution means input size decisions decide your cost.
  • A 4x int8 speedup on the same GPU or on-device phone CPU is often free accuracy-wise with quantization-aware training.
  • Serving glitches (wrong resize, channel order) silently destroy model quality at 10k QPS.
  • Monitoring on real image distributions (brightness, orientation, class priors) is exactly what drift detection needs.
Think about it

Mental model

Imagine

A food truck with a production-grade kitchen vs a cooking demo

Think of

CNN training vs production serving

The demo kitchen gets one perfect plate in an afternoon; the food truck serves 500 orders during lunch with identical quality, sub-60-second waits, and zero cross-contamination — that's the export/batching/quality-slop discipline.

Mechanics

How it works

  • Export the model to the serving format: TorchScript, ONNX, or a runtime-specific convert (TFLite, CoreML, TensorRT).
  • Quantize: FP32→INT8 via range calibration; QAT (quantization-aware training) preserves accuracy best.
  • Batch server: queue N requests, process together on GPU for throughput — latency bounds batch size.
  • Pre/post pipelines: decode, resize, normalize with EXACTLY the training stats; decode+map at the edge.
  • Monitor serving inputs (brightness, resolution histogram, L2 embedding distance) against training distribution.

The serving path from request to prediction.

Request
Preprocess
Batch
Infer
Monitor
Real world

In production

  • Cloud image tagging serves ONNX/TensorRT at a few thousand QPS with p95 < 60ms on GPU.
  • Android/iOS face-liveness uses TFLite/CoreML with INT8 quantized MobileNet — <20MB binaries.
  • Real-time video pipelines run YOLO-family on edge jetson/Raspberry hardware with fixed frame-skip budgets.
Guidelines

Best practices

  • Standardize the serving preprocessing (Lib-saving) as production code and unit-test it against training.
  • Benchmark latency percentiles, not just mean — p95 drives SLOs.
  • Prefer QAT for accuracy-critical quantized models; PTQ when benchmark says accuracy is fine.
  • Log input distribution stats (brightness, size) so a preprocessing config change is visible in dashboards.
Watch out

Common mistakes

  • Deploying FP32 because 'int8 sounds risky' — missing an easy 2-4x that often costs <0.5% accuracy.
  • Batching unbounded (latency spikes) or none at all (GPU idle, CPU trashing).
  • Different preprocessing in serving than training (RGB/BGR swap, /255 twice).
  • No input-distribution monitoring, so a camera config change silently toggles 'accuracy'.
Root cause analysis

Common issues

Error
Performance at 1 QPS is great; at 100 QPS, p95 quadruples
Root cause

Queueing/batch-interaction latency, or tensor conversion on the hot path

Fix

Add dynamic batching with a max-wait, prefetch/predecode, and move conversions off the critical path.

Error
Quantized model is 8% worse than FP32
Root cause

Naive post-training quantization on a sharp-minimum model

Fix

Quantization-aware training (QAT), per-channel calibration, and keep the first/last layers in FP16.

Error
Production accuracy differs from validation
Root cause

Serving preprocessing drift (resize mode, color, EXIF rotation) or input distro shift

Fix

Pin the exact preprocessing; add input-distribution/embedding-drift monitors at deploy.

When things break

Debugging tips

  • Diff the edge: feed the same image through notebook-pipeline and serving-pipeline; outputs must match to 1e-3.
  • Profile decode+resize: images decode is frequently the largest serving latency in practice.
  • Under load testing, watch p99 and queue depth, not average latency.
  • For edge, measure FIRST-inference cold-start too — model load may exceed your budget.
Interviews

Interview checkpoint

Practice

Hands-on lab

Benchmark Full vs Quantized Inference
30 min · advanced
In progress
Objectives
  • Build a small CNN and time full-precision inference
  • Simulate quantization (or use a framework's native API) and re-benchmark
  • Measure the accuracy-speed trade-off on a controlled problem

Benchmark Full vs Quantized Inference

30 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What is the primary benefit of quantizing a CNN to INT8?

2

Serving preprocessing must match training preprocessing exactly.

Go deeper

Official documentation

TensorRT developer guide
docs
ONNX Runtime docs
docs