APIs
Intermediate 14 min read APIs

Model SDK Patterns

One pattern, many providers

Model SDKs wrap HTTP in ergonomic clients. Learn the shared pattern — chat completions, streaming, tool schemas, embeddings — so switching providers or wrapping a gateway is a config change, not a rewrite.

SDKchat completionsstreamingclientprovidersabstraction
Prerequisites: HTTP & REST for AI
Before we start

What you will learn

  • Drive a provider SDK through its chat-completions and streaming interfaces
  • Abstract the provider behind an interface so swapping is cheap
  • Configure base_url to route through a gateway
Introduction

What is this?

Model SDKs give consistent Python/TypeScript clients over HTTP: you build a client, pass messages, get back completions, and opt into streaming. The chat pattern is near-standard across providers, so a small client interface lets you route to a frontier chat model, a cheaper model or a gateway — each behind the same call shape.

Motivation

Why do you need it?

  • You will switch models and providers as prices, quality and latency evolve; interfaces make that a config change.
  • Streaming is the same API with a flag; building on it unlocks token-by-token UX.
  • SDKs emit rich metadata (usage, latency) that feeds observability and cost tracking.
  • The AI Gateway pattern (the platform default via AI SDK) exposes one uniform client over many providers.
Think about it

Mental model

Imagine

A universal power adapter

Think of

A provider-agnostic client interface

Different sockets, one adapter: you plug your device in the same way anywhere. A thin client interface lets your service 'plug into' any model provider identically.

Mechanics

How it works

  • Instantiate a client with api_key, base_url and timeout; the SDK manages HTTP, retries and auth.
  • Send messages as a list of roles (system/user/assistant/tool) to a chat-completions endpoint.
  • Set stream=True to receive an iterator of delta chunks instead of one full JSON.
  • Provider responses include usage (prompt/completion tokens) the SDK exposes for cost tracking.
  • A gateway base_url lets the same SDK target routing, caching and rate limiting upstream.

Uniform client over providers and gateway.

Your service
SDK client
Gateway
Providers
Hands-on

Examples

A thin interface over chat completions

A thin interface over chat completions
from dataclasses import dataclass
from typing import Protocol, Iterator
 
@dataclass
class ModelResult:
    content: str
    usage: dict
 
class ChatModel(Protocol):
    def complete(self, messages, stream=False) -> ModelResult | Iterator[str]: ...
 
def make_client(provider: str, api_key: str) -> ChatModel:
    # returns a provider-specific adapter; swapping providers = swapping this factory
    ...

Define the shape you need (complete/stream), then implement it per provider. Callers stay provider-agnostic, and the gateway is configured via base_url.

Real world

In production

  • A customer-support bot serves a frontier chat model via a gateway and falls back to a cheaper model when the query classifies as simple.
  • A codegen tool streams completions token-by-token to the editor while logging usage for cost attribution.
  • A batch job pages through a cheapter-completions iterator with pagination/limits and writes structured outputs.
Guidelines

Best practices

  • Wrap the SDK behind your own interface; never call provider-specific methods deep into app logic.
  • Pass usage/latency from the response into your observability pipeline.
  • Configure base_url from env so the same code can target providers or a gateway.
  • Prefer streaming for interactive paths and buffered calls for batch jobs.
Watch out

Common mistakes

  • Hardcoding a provider's SDK calls across 40 files, making migration a fork.
  • Ignoring usage metadata and losing cost/time attribution.
  • Using a non-streaming call for a chat UI, then wondering why UX feels slow.
  • Assuming messages format is identical across providers — normalize roles early.
Root cause analysis

Common issues

Error
Response arrives but usage is None
Root cause

SDK version or provider omits usage metadata

Fix

Upgrade/pin the SDK or fetch usage from gateway headers/metrics.

Error
Model name not found when pointed at a gateway
Root cause

Gateway uses different model identifiers than the provider

Fix

Check the gateway's model list (e.g. curl the gateway models endpoint) and use its IDs.

Error
Chat works, streaming errors with 400
Root cause

Stream flag needs a provider-compatible call signature (e.g. stream_options)

Fix

Compare stream params and response iterator shape against the SDK docs.

When things break

Debugging tips

  • A model returns odd schemas — print the raw response once to compare with documented shape.
  • Streaming stops mid-response: check for interrupted iterator or a proxy buffering.
  • If messages look fine but the model ignores them, verify you sent the `messages` list, not a single string.
  • Test against the gateway's mock/echo provider first, then the real provider.
Interviews

Interview checkpoint

Practice

Hands-on lab

Build a Provider-Agnostic Chat Client
35 min · intermediate
In progress
Objectives
  • Define a ChatModel interface with complete/stream
  • Implement adapters for a mock provider and a real gateway without keys
  • Route by config with an offline fallback that returns deterministic output

Build a Provider-Agnostic Chat Client

35 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Why wrap a provider SDK behind your own interface?

2

Streaming completions use the same SDK method as non-streaming with a flag.

Go deeper

Official documentation

AI SDK (Vercel) chat interface
docs
OpenAI Python SDK
docs