Calling LLM APIs
The durable interface to intelligence
LLM products consume hosted APIs: model selection, inputs (messages/tools/params), outputs (content, usage, finish reason), errors and rate limits. This concept wires a chat call with production hygiene.
What you will learn
- Describe the standard chat-API contract
- Read usage fields and finish reasons
- Handle common API failure modes
What is this?
Hosted LLM APIs take a list of messages (system/user/assistant/tool) plus parameters and return generated content plus usage metadata. The SDK abstracts transport and JSON; a gateway (like the platform's AI Gateway default) gives you provider routing, retries and consistent observability without per-vendor glue.
Why do you need it?
- One standard contract spans providers; SDK + gateway hide the differences.
- usage fields drive cost tracking and budgets.
- finish_reason tells you whether you truncated, hit a stop, or a tool call.
- Rate-limit/retry behavior is a production requirement, not a nicety.
Mental model
Placing an order at a kitchen
An LLM API call
You hand the kitchen a ticket (messages + specs). The kitchen sends back the dish (content) plus a receipt (usage — tokens/cost) and a note if it could only make part of it. Order handlers (SDKs) ensure typos don't sink the whole dinner.
How it works
- Client sends messages + model + parameters (temperature, max_tokens, stream).
- Provider runs the autoregressive loop internally and returns text, finish_reason and usage.
- Streaming variants yield token chunks incrementally.
- Framework wrappers (like AI SDK) add tool calling, structured outputs and gateway defaults.
- Proxy/gateway layers centralize keys, routing, retries and logging.
One chat-call lifecycle.
Examples
Minimal chat call (AI SDK style)
import { generateText } from "ai"; const { text, usage, finishReason } = await generateText({ model: "openai/gpt-4o-mini", // routed via AI Gateway system: "You are a concise assistant.", prompt: "Summarize the diff in one line.", temperature: 0.2,}); console.log(text);console.log("tokens:", usage.inputTokens, usage.outputTokens);console.log("finish:", finishReason);The gateway default is already wired in the AI SDK: one model string, standard metadata back.
In production
- Serverless functions call via the gateway for key isolation and observability.
- Batch jobs loop calls with backoff and per-job budgets.
- Failure handling inspects finish_reason to distinguish truncation from tool_intent.
Best practices
- Store provider keys in platform env vars; never in the client bundle.
- Centralize LLM access behind one client module (easier routing/logging later).
- Log model, params and usage per call for cost/quality analysis.
- Set sensible timeouts and exponential backoff on rate limits.
Common mistakes
- Putting API keys in frontend code or public env.
- Ignoring usage fields (cost surprises).
- Treating every non-200 as retryable (400s are client bugs — don't retry).
- Swallowing finish_reason — truncated generations slip through silently.
Common issues
401 Unauthorized from providerKey missing/wrong scope in the gateway env
Verify server env var, provider project scoping, and gateway role.
Answer cuts off mid-sentencemax_tokens hit before the model finished
Check finish_reason, raise output budget or restructure the task.
Rate-limited bursts in prodNo backoff or concurrency too high
Add retry with jittered backoff and client-side concurrency caps.
Debugging tips
- 401/403 → key or scope; 429 → backoff; 400 → request shape; never retry 400.
- Truncated answers → check finish_reason length, raise max_tokens or chunk the task.
- Slow first calls → cold gateway/provider; warm or accept, but track p50/p95.
Interview checkpoint
Hands-on lab
- Make a chat completion via a hosted API or gateway
- Inspect usage and finish reason
- Handle a failure with a retry
First Chat Call with Metadata
30 min · beginner
Quick quiz
Which field tells you a generation stopped early due to token limits?
HTTP 400 responses should usually be retried with backoff.