Choosing an Inference Engine
The serving library is a product decision
vLLM, TGI, TensorRT-LLM and SGLang differ in batching, quantization support, speculative decoding and hardware optimization. The right engine matches your hardware, your latency profile and your feature needs — benchmark on your workload, not the README.
What you will learn
- Compare the major inference engine families
- Match engine features to hardware and workload
- Benchmark engines on tokens/sec and TTFT
What is this?
An inference engine turns raw weights into a serving runtime: continuous batching, quantization kernels, speculative decoding and a chat-compatible API all live here. vLLM is the open default; TensorRT-LLM wins on NVIDIA-only peak performance; TGI has enterprise polish; SGLang shines on complicated attention control.
Why do you need it?
- Engine choice moves tokens/sec 30-200% with the same GPU.
- Feature gaps (quantization schemes, structured outputs) can veto an engine.
- Engines are deployable units: swapping them changes ops posture, upgrade cadence and support.
Mental model
Inference engine
A car's drivetrain
Same engine block (weights), different gearboxes (engines/servers). The gearbox decides fuel economy, acceleration and towing (throughput, TTFT, long context).
How it works
- Engine loads weights into GPU-optimized kernels and exposes an OpenAI-compatible endpoint.
- Batching, cache and decoding policies are engine internals; most expose a few knobs.
- Some (TensorRT-LLM) compile a plan per model/hardware, others (vLLM) JIT optimized CUDA.
- Feature velocity differs: new quantization, spec-decode and decoding strategies land at different speeds.
Engine sits between weights and API.
In production
- An NVIDIA-only shop peaks with TensorRT-LLM via Triton; a startup defaults to vLLM for velocity.
- A structured-output-heavy app picks SGLang for its JSON/grammar scheduling.
- A mixed AMD/Intel + NVIDIA fleet needs a portable engine like vLLM to avoid per-hardware compiles.
Best practices
- Benchmark engines on YOUR prompt/out length mix at YOUR concurrency.
- Verify quantization + spec-decode feature support before committing.
- Prefer the most-maintained engine for your GPU generation; test upgrades in canary.
- Keep the API compatible (OpenAI) so engine swaps stay client-invisible.
Common mistakes
- Picking an engine on a paper benchmark of a different model/dataset.
- Running a compiled-plan engine without re-planning after model changes.
- Ignoring hardware support (some engines are NVIDIA-only; your fleet matters).
Debugging tips
- Same weights, different tokens/sec → engine kernels/scheduling; re-benchmark honestly.
- OOM after engine switch → KV-cache defaults differ; align concurrency settings.
- Weird outputs after swap → decoding/temperature defaults or tokenizer mismatch between engines.
Interview checkpoint
Quick quiz
Which factor often disqualifies an engine outright?
All inference engines behave identically with the same weights.