LLM Threat Models
Know the enemies
A threat model enumerates who can attack your LLM system and how: injection, data theft, tool abuse, denial of service, jailbreaks. Naming the threats is the first and most valuable security step.
What you will learn
- Enumerate LLM-system attack surfaces
- Map classes of LLM threats
- Prioritize mitigations by risk
What is this?
An LLM threat model identifies the asset (user data, tools, system instructions), the attacker (users, web content, RAG sources, insiders), and the paths: prompt injection, jailbreaks, exfiltration over tools, denial via cost/latency, and training-data leakage. Risk = impact × likelihood guides which mitigations ship first.
Why do you need it?
- You can't defend what you haven't named.
- Attack surfaces multiply with tools, memory and RAG.
- Threat modeling is cheap; remediating incidents is not.
- The model is the least of the attack surface — pipes and tools are worse.
Mental model
A castle's defense plan
Threat model
Before building walls, the lord maps who attacks: sieges (jailbreaks), treasons (insiders), tunnels (injection), supply cutoffs (DoS). Each threat gets a wall, a gate, a moat sized to the risk. Building towers first and forgetting the moat is the classic LLM mistake.
How it works
- Define assets: data, instructions, tool capabilities.
- Enumerate entry points: user chat, doc uploads, web fetch, RAG.
- List attacker classes: external, malformed content, insiders.
- Map threats per surface: injection, exfil, DoS, abuse.
- Rank by impact × likelihood; assign mitigations.
Threat map.
In production
- Security review gates for any new tool or RAG source.
- Red-team runbooks test the model's stated threats.
- Risk registry tracks open mitigations for audit.
Best practices
- Update the model whenever new tools/sources ship.
- Treat every model-visible byte as attacker-controlled.
- Focus on tools and pipes: they amplify damage.
- Cost-control is part of the model (DoS by budget).
Common mistakes
- Threat models that ignore injected content (docs/pages).
- Treating the model as 'secure because it's an API'.
- Writing the model once and never re-gating new features.
Common issues
Threat appeared not in the modelModel stale
Tie threat review to the feature-launch gate.
Mitigations untraceable to threatsNo mapping
Keep a 1:1 threat→control→test inventory.
Debugging tips
- New tool → new threat; add to the model before launch.
- Incident repeats → model missed the path; backfill.
- Mitigations untested → red-team the named threats.
Interview checkpoint
Hands-on lab
- Inventory assets, entry points and boundaries
- Enumerate threats per entry point
- Rank threats and assign mitigations
Mini Threat Model in Three Tables
25 min · intermediate
Quick quiz
The largest attack surface in an agentic system is…
Fetched web content is trusted input.