TorchServe

• Package a PyTorch model into a .mar archive
• Write a handler for preprocessing and inference
• Serve and health-check the model via TorchServe APIs

• It removes the 'write my own serving glue' tax around PyTorch models.
• Batching, warm-up and worker scaling are built-in, not hand-rolled.
• Model archives make deployment a file artifact, versionable in the registry.
• One engine handles many models with management APIs.

You bring the dough recipe (handler) and ingredients (model); the bakery provides the ovens, the batch schedule (batching), preheating (warm-up) and the shopfront (API). Your job is the recipe, not building an oven.

• You produce a .mar archive: serialized model, handler code, an index.json manifest, configs.
• torchserve --start loads archives; the frontend API accepts requests.
• Workers run your handler: preprocess → model forward → postprocess.
• Requests can be batched per config; warm-up primes workers before traffic.
• A management API adds/removes versions while workers keep serving.

• A vision team serves a quantized ResNet as one .mar; batching handles bursty inference.
• Two model versions share one TorchServe instance via the management API during rollout.
• Warm-up scripts prime the transformer backbone so first requests aren't slow on cold workers.

• Keep preprocessing deterministic and inside the handler, not the client.
• Set batching with a max delay so latency doesn't balloon at low load.
• Register .mar files in the model registry with their hash.
• Use warm-up for heavy models; cold starts otherwise penalize the first requests.

• Baking client-side preprocessing assumptions into the handler and breaking them at deploy.
• Turning batching on without a max wait, spiking latency under sparse traffic.
• Not versioning .mar archives, so redeploys drift from the validated artifact.
• Ignoring worker count — one worker serializes all your throughput.

• First call slow, later fast — the cold worker is running warm-up; add it explicitly.
• Batch queue grows — workers may be stuck on a slow request; check per-worker logs.
• Archive mismatch — compare the .mar hash in the registry to the one TorchServe loaded.