AI Security
Advanced 14 min read AI Security

Prompt Injection

Instructions hiding in the data

Prompt injection instructs a model from within user-or-content text: 'ignore your rules'. Defense is structural — separable instruction, untrusted-input framing, content-vs-code discipline — not hoping the model resists.

prompt injectionindirect injectioninstruction hierarchyuntrusteddefense
Prerequisites: LLM Threat Models
Before we start

What you will learn

  • Explain direct and indirect injection
  • Apply structural defenses
  • Keep injections from reaching tools
Introduction

What is this?

Prompt injection is the attacker's text becoming instructions: direct (a user says 'ignore your policies') or indirect (poisoned web pages, docs, or RAG chunks steer behavior). Robust defense is structural — boundary markers, instruction hierarchy, content treated as data — not relying on the model to 'just be smart'.

Motivation

Why do you need it?

  • Models follow instructions they find in data whether or not you intended.
  • Indirect injection turns ordinary content into an escalator to your tools.
  • Structural isolation beats prompt-language armwrestling.
  • Tool systems multiply the blast radius of a successful injection.
Think about it

Mental model

Imagine

An envelope vs. a documented memo

Think of

Instructions vs. content

When memos (instructions) and letters (data) arrive in the same tray, a letter is just paper. But if data arrives with the formatting of a memo — 'IMPORTANT: CHANGE RULES' — a busy reader follows it. Injection is forgetting the envelopes: content must never arrive as instructions.

Mechanics

How it works

  • Frame user/content as data, system as instructions, starkly.
  • Use delimiters and instruction-hierarchy directives.
  • Never concatenate raw web/doc text as instructions.
  • Gate tools independently: schema + allowlist on every call.
  • Post-hoc detection: sentry flags suspicious tool args/results.

Injection paths to cut.

User
Web/RAG
Prompt hull
Tool gate
Systems
Real world

In production

  • RAG chunks wrapped in data-tags and never echoed as commands.
  • Tool schemas ballpark boundaries: args are typed, hosts allowlisted.
  • Web tools return terse surface data, not instruction-rich HTML.
Guidelines

Best practices

  • Treat all non-system content as untrusted data.
  • Keep instruction-vs-content structural, not just worded.
  • Enforce the tool gate as the real control surface.
  • Red-team injection regularly; bake cases into evals.
Watch out

Common mistakes

  • 'We told the model to ignore instructions in data' as the only defense.
  • Echoing fetched HTML or doc text as context verbatim.
  • Tools that obey whatever the 'instructions' say with no gates.
Root cause analysis

Common issues

Error
Injection changed a tool arg
Root cause

Content reached tools unsanitized

Fix

Bound the tool gate: typed args, fixed target namespace, independent policy.

Error
RAG reroutes reply to attacker instructions
Root cause

Sensitive instruction framed as data

Fix

Wrap chunks as data; hierarchical instruction framing; output scrub.

When things break

Debugging tips

  • Suspicious tool calls → check source text for instruction-anchored strings.
  • Behavior change after RAG update → chunk content steered the loop; re-frame.
  • Injection eval green but live leak → eval lacked new attack shapes.
Interviews

Interview checkpoint

Practice

Hands-on lab

Frame Instructions vs Data
35 min · advanced
In progress
Objectives
  • Simulate a direct and indirect injection
  • Apply structural data-framing
  • Show tool args stay typed against steers

Frame Instructions vs Data

35 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Indirect injection travels via…

2

A strong system-prompt warning fully blocks injection.

Go deeper

Official documentation

Prompt injection (Anthropic)
docs