Cost Optimization
Beginner 10 min read Cost Optimization

Token Economics

Every word has a price

You pay per token — prompt + completion — with output tokens often priced several times higher than input. System prompts, retrieved context and long chat histories are the hidden spend; cache reads and input discount tiers change the math.

token pricinginput vs outputcost per tokencache reads
Prerequisites: Scaling models
Before we start

What you will learn

  • Compute cost from prompt and completion tokens
  • Recognize the biggest hidden drivers
  • Account for cache and input-tier discounts
Introduction

What is this?

LLM bills are token-based: cost = (prompt_tokens × prompt_price) + (completion_tokens × completion_price), often with output 3-5x more expensive than input. Because your app pays for sent tokens too, prompt bloat and context accumulation are real line items.

Motivation

Why do you need it?

  • A 100-line system prompt is paid on every request, every time.
  • Completion tokens are the pricey ones by design — terse outputs save real money.
  • Retrieval dumps swallow budgets invisibly before the model even answers.
Think about it

Mental model

Imagine

Token billing

Think of

Postage by envelope size

A short receipt (output) is cheap; a thick padded mailer (gigantic prompt) costs more and you pay it on both legs.

Mechanics

How it works

  • Providers price input vs output per-million tokens differently.
  • Prices scale with model size — frontier chat models cost far more per token than fast ones.
  • Cache reads bill at a deep discount (often ~10% of input price).
  • Your true cost is tokens × price; providers' usage fields make it measurable per call.

Your bill's anatomy.

Prompt
Completion
Cache
Total
Hands-on

Examples

Reading your own bill

Reading your own bill
prompt_tokens = 4200      # system + context
completion_tokens = 240
price = {"input": 0.15, "output": 0.60}  # per 1M tokens
cost = (prompt_tokens * price["input"] + completion_tokens * price["output"]) / 1e6
print(f"cost: {round(cost, 4):.4f} for one request")

That single retrieval-heavy call cost more than 100 short ones — the ratio is where the surprises live.

Real world

In production

  • A support bot trims history to a rolling window to cap prompt growth.
  • A translation service picks a cheaper model for short, unambiguous strings.
  • Semantic cache turns repeated retrieval-heavy prompts into discounted reads.
Guidelines

Best practices

  • Track tokens per request per endpoint, not just totals.
  • Keep system prompts lean; prune returned context to what the model needs.
  • Prefer cache-friendly structures so recurring prefixes hit the discount.
  • Alert when tokens-per-request drifts up on a stable traffic pattern.
Watch out

Common mistakes

  • Ignoring prompt cost because output price looks bigger on paper.
  • Pasting entire RAG paragraphs into the prompt unchanged.
  • Unbounded multi-turn history that compounds forever.
When things break

Debugging tips

  • Cost up, traffic flat → check average prompt tokens and cache-hit rate.
  • 'Free' chattiness: more tokens returned than your product needs — set max tokens.
  • If one endpoint dominates spend, dump its token distribution by request.
Interviews

Interview checkpoint

Practice

Hands-on lab

Read a Token Bill Correctly
25 min · beginner
In progress
Objectives
  • Compute cost from prompt and completion tokens
  • Spot the hidden drivers in a mock usage log
  • Compare price tables across two models

Read a Token Bill Correctly

25 min · beginner

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which usually costs more per token?

2

Cache reads are billed at a discount compared to fresh prompt tokens.

Go deeper

Official documentation

OpenAI pricing
docs
Anthropic token usage
docs