Tokenization
Beginner 10 min read Tokenization

Token Costs & Counting

The accounting layer of AI

Billing is per token in and out. Counting tokens right — input vs output, cache hits, replication per retry — is the difference between an invoice you expect and a bill that shocks.

token costbillinginput tokensoutput tokenspricingtoken counting
Prerequisites: BPE
Before we start

What you will learn

  • Compute per-call token costs
  • Track input vs output token usage
  • Budget for retries, caching and replication
Introduction

What is this?

Providers bill by token volume: input tokens (prompt) and output tokens (completion) at different rates, sometimes with cached-input discounts. Correct counting multiplies tokens by price with per-provider modifiers, plus retries and background jobs — where hidden multipliers hide.

Motivation

Why do you need it?

  • Cost = tokens × price; both halves are controllable.
  • Output tokens usually cost more per token than input.
  • Retries, cache misses and tool loops silently multiply spend.
  • Dashboards must distinguish token types to find waste.
Think about it

Mental model

Imagine

A metered taxi with rush-hour surge

Think of

Token billing

The metre counts distance (tokens); input distance is cheaper per km than the output stretch, and rerouting the same street twice (retry) adds up fast. Planning the route (a tight prompt) is how you keep the metre honest.

Mechanics

How it works

  • Read usage fields (inputTokens/outputTokens) per call from SDK responses.
  • Multiply by the provider's per-1k-token prices for input and output separately.
  • Add cache-aware pricing if the platform exposes cached-input tiers.
  • Sum across retries, tool round-trips and background jobs.
  • Feed per-call numbers into a cost dashboard for trends.

From tokens to spend.

Call
Prices
Rows
Aggregate
Alert
Hands-on

Examples

A tiny cost engine

A tiny cost engine
IN_PRICE = 0.15   # $ per 1M input tokens
OUT_PRICE = 0.6    # $ per 1M output tokens
 
def cost(input_tokens, output_tokens):
    return (input_tokens / 1_000_000 * IN_PRICE
            + output_tokens / 1_000_000 * OUT_PRICE)
 
print("call cost:", round(cost(2_000, 400), 5), "$")
print("1000 such calls:", round(1000 * cost(2_000, 400), 2), "$")

Inputs dominate for chat prompts; outputs dominate when the app writes long content.

Real world

In production

  • Attach a budget to every internal API user so runaway scripts can't spike spend.
  • Cache repeated prompts (prompt KV caching) to discount input tokens.
  • Log per-call usage with a source/feature tag for chargeback analytics.
Guidelines

Best practices

  • Always capture usage fields; never estimate from characters.
  • Tag calls (feature, tenant, env) on ingest, not at billing time.
  • Set hard per-tenant budgets with blocking alerts.
Watch out

Common mistakes

  • Forgetting output tokens are billed at a premium.
  • Billing retries as if once — each retry bills input+output again.
  • Rounding token counts to characters for budgeting.
Root cause analysis

Common issues

Error
Spend spike with stable traffic
Root cause

Retry storms or large cached prompts without discount

Fix

Rate-limit retries, cap max_tokens, enable cached-input pricing.

Error
Cost per call differs widely
Root cause

Prompt length variance across requests

Fix

Normalize prompts; surface top costly prompt shapes in dashboards.

When things break

Debugging tips

  • Invoice jump but no user growth → look at retries/background jobs/tool loops.
  • High input cost + low output → fat prompts; trim context, enable prompt caching.
  • Per-feature tags make the spike answer obvious.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which token type usually commands a higher per-token price?

2

A retried call re-bills the full prompt tokens.

Go deeper

Official documentation

OpenAI pricing page
docs
Anthropic pricing page
docs