Model Deployment
Advanced 16 min read Model Deployment

Inference Optimization

Cheaper, faster, still good enough

Making inference fast and cheap without wrecking quality: quantization, batching/kernel fusion, smaller models (distillation), pruning, and caching. You optimize the cost/latency/quality triangle with an eval harness measuring all three.

quantizationbatchingdistillationpruningkernellatencythroughputGPU utilization
Prerequisites: Model Serialization Model Serving
Before we start

What you will learn

  • Quantize models (INT8/FP16) and measure quality impact
  • Use dynamic batching and parallel kernels to lift throughput
  • Evaluate optimizations against quality + latency + cost together
Introduction

What is this?

Inference optimization cuts cost and latency while keeping quality within a certified band. Toolkit: quantization (FP16/INT8 — smaller weights, faster matmuls), distillation (train a small student to mimic a big teacher), pruning (drop near-zero weights), kernel/compiler tricks (ONNX Runtime, TensorRT, vLLM, torch.compile) and dynamic batching (fill GPU idle slots). Every change is validated by an eval harness that scores quality on the SAME eval set as pristine inference.

Motivation

Why do you need it?

  • $/prediction and p95 latency decide whether a feature is viable at all.
  • Smaller models serve more QPS per GPU and cut CO2/cost per inference.
  • Optimized serving survives peak seasons without a fleet doubling.
  • Good-enough quality within the cert band beats perfect-but-unaffordable.
Think about it

Mental model

Imagine

A 100-seat restaurant cutting the kitchen cost

Think of

Inference optimization

You'd run the same menu (quality band), but: buy a faster oven (kernel), portion frozen prep (quantized weights), train a junior chef on the senior's plating (distillation) and seat parties together instead of one-by-one (batching). Fire someone's signature dish and the math collapses — check quality, not just cost.

Mechanics

How it works

  • Baseline: measure p50/p95 latency, QPS, max batch, and quality metrics on the eval set — pure CPU/GPU native first.
  • Quantize: FP16 (easy), INT8 (requires calibration; watch accuracy cliffs).
  • Distill: student model trained on teacher logits; quality check vs band.
  • Fuse + compile: ONNX/TensorRT/vLLM kernels fuse ops and use optimized matmuls.
  • Batch: dynamic batching queues requests into fixed-size batches for max GPU utilization.

Optimization levers with their quality guardrails.

Baseline
Quantize
Fuse/compile
Distill
Certify
Real world

In production

  • LLM serving: vLLM/continuous batching is table stakes for token economics.
  • Edge/CPU models: INT8 quantization for on-device and small instances.
  • High-QPS tabular: ONNX Runtime dynamic batching on modest CPUs.
Guidelines

Best practices

  • Certify on a FIXED eval set and diff against pristine-inference scores — every lever, a number.
  • Profile before optimizing: if matmul is 80% of time, quantize; if it's I/O, cache instead.
  • Quantize to FP16 first (free win), then INT8 only where the band allows.
  • Decouple batching from requests — always serve within a latency budget per batch size.
Watch out

Common mistakes

  • Optimizing without a quality harness — 'seems fine' is how silent 2% regressions ship.
  • Batching at 10 requests when p95 blows — latency and throughput must be traded explicitly.
  • INT8 on a model with sensitive tails (long-tail classes) without per-class check.
  • Measuring on a tiny benchmark set so the 'win' is noise.
Root cause analysis

Common issues

Error
INT8 accuracy collapse
Root cause

Calibration set unrepresentative or sensitive outlier activations

Fix

Fix calibration, use per-channel quantization, or stay FP16 for the sensitive tail.

Error
Throughput up but p95 terrible
Root cause

Batches too big / queueing unmanaged

Fix

Cap batch size by p95 budget; batch across requests, never within a request, past the SLA.

Error
Compiled build slower than eager
Root cause

Kernel fusion overhead for small models/short inputs

Fix

Profile per-op; for small models keep native runtime and spend effort on batching instead.

When things break

Debugging tips

  • Profile per-operator: quantized/compiled builds should change the matmul wall, not mysteriously help I/O.
  • Compare quantized vs original logits on the eval set; a big KL gap predicts quality regressions in production.
  • Watch GPU utilization and batch under a QPS load test — idle GPU = immature batching.
  • A/B in shadow: deploy optimized behind traffic and diff scores vs the reference endpoint.
Interviews

Interview checkpoint

Practice

Hands-on lab

Measure the Quantization Win
35 min · advanced
In progress
Objectives
  • Quantize a tree model to INT8 via ONNX and compare latency
  • Diff model outputs before/after quantization
  • Certify a quality metric is within band

Measure the Quantization Win

35 min · advanced

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The safe order when optimizing inference quality-wise is:

2

FP16 quantization is generally a low-risk, high-value first step.

Go deeper

Official documentation

ONNX Runtime optimization
docs
TensorRT docs
docs