Providers vs Self-Hosting
A decision matrix, not a religion
Managed providers abstract GPUs, scaling and retries behind a key; self-hosting puts them on you but lowers marginal cost and keeps data in-house. Choose with a decision matrix of latency, data, cost, and team capacity.
What you will learn
- Score providers vs self-hosting on five axes
- Estimate total cost of ownership for both
- Design a hybrid routing escape hatch
What is this?
Providers (Anthropic, OpenAI, Azure, Bedrock, cloud vertex endpoints) rent you inference per token with wide model choice and near-zero ops. Self-hosting (vLLM, TGI, TensorRT-LLM on your GPUs) fixes your per-token cost but puts capacity planning, patching and reliability on your team.
Why do you need it?
- License and privacy clauses differ — many enterprise models require on-prem.
- Unit cost flips with scale: high volume funds the GPU investment fast.
- Provider outages become your outages unless you have a second provider or a self-hosted fallback.
Mental model
Providers
Software-as-a-service
You use them for speed; the dependency on their uptime and pricing is the rent you pay.
How it works
- Providers run fleets of GPUs and serve many tenants; your cost reflects amortized hardware plus margin.
- Self-hosting buys/rents GPUs and runs an inference server tuned to your workload.
- A gateway can treat providers and self-hosted replicas as interchangeable upstreams.
- Cold-start differs: providers scale instantly-ish; self-hosted replicas must have capacity pre-provisioned.
TCO breaks even at your traffic volume.
In production
- A fintech self-hosts an 8B model for PII-heavy prompts, keeps the frontier model managed.
- A platform bills teams internally; the router sends 80% of tokens to a cheap self-hosted model.
- An enterprise runs Anthropic for creative work and vLLM for high-frequency classification.
Best practices
- Write your own client abstraction so upstream swaps are a config change.
- Keep a second provider or a tiny self-hosted fallback for critical paths.
- Track actual tokens/sec and price per 1K on every upstream.
- Pre-provision one warm self-hosted replica before cutting over.
Common mistakes
- Migration to self-hosting without measuring the org's capacity to run GPUs.
- Single-provider lock-in with no route back.
- Buying GPUs for bursty traffic that idles 80% of the time.
Debugging tips
- Same model, provider vs self-host, same temp: compare token-by-token logprobs to catch version drift.
- Retry/provider 429 vs self-host 503 usually signal capacity, not code.
- Watch cold-start: a 5-minute model load will look like a bad timeout curve.
Interview checkpoint
Hands-on lab
- Model provider vs self-host unit costs
- Compute a break-even volume from real-shaped numbers
- Sensibility-check GPU amortization assumptions
Cost-per-Token Calculator
30 min · intermediate
Quick quiz
Which factor most strongly pushes enterprises to self-host?
Provider pricing is linear, so scale never changes the comparison.