Inference Optimization
Intermediate 11 min read Inference Optimization

Latency Budgets

Slicing latency until it fits the SLO

A latency budget breaks the user-visible deadline into measurable slices — network, gateway, queue, prefill, decode, app logic — so you tune the slice that hurts, not the whole stack. TTFT and inter-token latency are the two numbers that matter.

latency budgetTTFTinter-token latencypercentileSLO
Prerequisites: Streaming Responses
Before we start

What you will learn

  • Decompose end-to-end latency into slices
  • Set TTFT and token-rate targets
  • Decide which slice to optimize first
Introduction

What is this?

A latency budget is a pre-agreed allocation of time across every stage a request crosses, summing to the p95 the product can tolerate. For LLM apps the two dominant slices are TTFT (time to first token) and inter-token latency (time between subsequent tokens), which together set perceived responsiveness.

Motivation

Why do you need it?

  • You cannot optimize what you haven't timed — budgets make the SLO concrete.
  • TTFT and inter-token latency are traded off differently: batch hurts TTFT but helps throughput.
  • A budget stops over-engineering: hit p95 2000ms with 1.8s in the model, don't tune DNS.
Think about it

Mental model

Imagine

Latency budget

Think of

A film's run time

Every scene (stage) has a sticky budget; if the director overruns in Act 2, Act 3 must be trimmed or the whole film moves.

Mechanics

How it works

  • Pick an SLO: e.g. p95 first token < 1200ms, p95 total < 6000ms for a 2K-token answer.
  • Instrument every hop — client, edge, gateway, queue-wait, prefill, decode, app render.
  • Bills of the slices must be under the SLO with headroom.
  • When a slice breaks budget, fix that slice instead of complaining about the model.

A 6s budget broken into slices.

Client→GW
Queue
Prefill
Decode
App/final
Hands-on

Examples

Checking a budget with a script

Checking a budget with a script
ttft_ms = 850
token_rate = 22          # tokens/sec
answer_tokens = 1200
tail_ms = 300
total_ms = ttft_ms + (answer_tokens / token_rate) * 1000 + tail_ms
print('p95 estimate ms:', round(total_ms))
print('within 6000ms budget:', total_ms < 6000)

The decode slice dominates; raising tokens/sec or cutting answer length is where the budget is won.

Real world

In production

  • A chat product sets SLO TTFT 1s and measures every deploy against it.
  • An agent loop budgets 500ms per tool round trip so a 5-step agent finishes in 2.5s.
  • An API sla budgets p95 10s total for long generations, streaming as it goes.
Guidelines

Best practices

  • Measure percentiles, not averages — the tail is what users feel.
  • Set separate TTFT and total SLOs; they have different levers.
  • Re-budget after model upgrades; a faster engine changes the split.
  • Log each slice per request (tracing IDs) so regressions are attributable.
Watch out

Common mistakes

  • Only measuring total time, so the real culprit (queue or decode) is invisible.
  • Optimizing prefill to death when decode dominates 80% of the budget.
  • A budget whose slices add up to 110% of the SLO — no headroom.
When things break

Debugging tips

  • TTFT over budget → queue depth or prefill; decouple before blaming the model.
  • Tokens/sec low → decode bound (batch, model size, quantization).
  • Total over but slices under → missing slice (e.g. client-side processing).
Interviews

Interview checkpoint

Practice

Hands-on lab

Slice a Latency Budget
25 min · intermediate
In progress
Objectives
  • Model an end-to-end latency budget numerically
  • Attribute the tail to the dominant slice
  • Propose a re-budget that meets the SLO

Slice a Latency Budget

25 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which slice usually dominates a long chat completion?

2

A single 'total time' SLO is enough to tune latency.

Go deeper

Official documentation

Latency at Anthropic
docs
OpenAI latency guide
docs