Model Serving
Beginner 10 min read Model Serving

Streaming Responses

First token fast, rest as they happen

Streaming sends tokens to the client as they are generated over SSE instead of after the full completion. It cuts perceived latency to the first token, keeps long generations from timing out, and is table stakes for chat UIs.

streamingSSEserver-sent eventsTTFTtoken stream
Prerequisites: Serving Strategies
Before we start

What you will learn

  • Explain TTFT vs total latency
  • Implement SSE streaming end to end
  • Handle partial token buffers correctly
Introduction

What is this?

Streaming serves text incrementally: the caller opens a connection (usually SSE), receives a chunk per token (or per delta), and renders it live. Non-streamed calls return a single body once the whole completion is ready.

Motivation

Why do you need it?

  • Users perceive a 300ms TTFT + flowing text as far faster than a 5s blank spinner.
  • Long generations (agents, article drafts) otherwise risk client timeouts.
  • You can cancel early (the famous stop mid-generation), saving tokens.
Think about it

Mental model

Imagine

Streaming

Think of

A voice call

You hear sentences arrive live; the alternative is a transcript that appears only when the call ends.

Mechanics

How it works

  • The client opens an SSE (or chunked HTTP) connection and reads events.
  • Server emits one event per delta; chat SDKs accumulate deltas into the visible message.
  • When generation ends, send a [DONE] marker so clients close cleanly.
  • Streams can also carry tool-call fragments, usage snapshots and signal names.

SSE lifecycle.

Connect
Delta
Interrupt
DONE
Hands-on

Examples

A minimal SSE handler shape

A minimal SSE handler shape
curl -N -X POST https://api.provider.dev/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
    "model": "fast-model",
    "messages": [{"role":"user","content":"write 3 haiku"}],
    "stream": true
  }'
 
data: {"delta":"the "}
data: {"delta":"moon "}
data: {"delta":"over "}
data: [DONE]

Each data: line is one delta; most SDKs auto-accumulate. [DONE] is the well-known termination marker.

Real world

In production

  • A chat UI renders streaming responses with a blinking cursor.
  • An agent terminal shows tool calls arrive live and lets the user abort.
  • A search page streams the answer above the links while links load underneath.
Guidelines

Best practices

  • Set stream:true and expose TTFT (first chunk time) as a metric.
  • Encode deltas UTF-8-safe; tokens may split multi-byte characters.
  • Add client- and server-side timeouts aligned to generation length.
  • Send heartbeat/keep-alive so idle streams aren't killed by proxies.
Watch out

Common mistakes

  • Buffering the whole stream server-side before forwarding — you lose the point.
  • Not handling reconnection: a dropped stream corrupts the UI mid-render.
  • Using the raw model stream but failing to strip control/thinking tokens for end users.
When things break

Debugging tips

  • TTFT slow but streaming smooth = server queueing, not transport.
  • Chunks arriving in bursts = buffering layers (proxy, framework) — check flush/stream settings.
  • Partial multi-byte characters: join deltas before slicing, never per-chunk.
Interviews

Interview checkpoint

Practice

Hands-on lab

Parse an SSE Token Stream
25 min · intermediate
In progress
Objectives
  • Feed a fake SSE response through a parser
  • Accumulate deltas into a complete message
  • Handle the [DONE] terminator

Parse an SSE Token Stream

25 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What does TTFT stand for?

2

Streaming increases total token generation time.

Go deeper

Official documentation

OpenAI streaming guide
docs
MDN Server-Sent Events
docs