Failure Handling
Fail by design
Agents fail in recognizable ways: tool errors, exhausted caps, divergence, hallucinated success. Failure handling makes each failure explicit — detect, classify, retry-or-degrade, and report honestly with usable partial state.
What you will learn
- Classify agent failures
- Apply retry, degrade, and fallback responses
- Report honest partial state on abort
What is this?
Failure handling predefines the response to every likely failure class: transient tool errors (retry with backoff), cap exhaustion (degrade scope or report partial), divergence (detect and re-plan or stop), and hallucinated success (verify evidence, fail honestly). Each failure is logged, classified, and converted to a user-facing, actionable outcome.
Why do you need it?
- Unhandled failure classes are the top source of agent incidents.
- Classification selects the response: retry vs degrade vs abort.
- Honest partial states beat invented completions.
- Live failure telemetry feeds evals and design fixes.
Mental model
Airline delay procedures
Failure handling
A delay has a playbook: rebook (retry), downgrade class (degrade), or inform with options (honest abort). No one pretends the flight is on time — that'd compound the failure. You get status, alternatives, and a path, because the procedures were written for the failure that is happening.
How it works
- Detect failure classes: transient, cap, divergence, fake-success.
- Classify on exit: code + context into telemetry.
- Respond: retry (transients), degrade (scope cut), fallback model/route, abort.
- Verify before reporting done; evidence gate at terminal.
- Compose the user-facing message: what failed, what's done, what to do.
Failure taxonomy → response.
In production
- Ops runbooks degrade gracefully when a probe times out.
- Consumer assistants report partial search results with retry hints.
- Eval harnesses verify aborts match expected failure classes.
Best practices
- Write a response for every failure class before shipping.
- Never report done without evidence (treat terminal as a gate).
- Degrade scope rather than silently bluff weaker output.
- Feed all failure telemetry into the eval suite.
Common mistakes
- Status quo: crash → empty response.
- Bluffing: returning optimistic 'done' to avoid the awkward report.
- Retrying divergence four times before realizing it's not transient.
Common issues
Failures multiply (fan-out of half-work)Degrade triggered writes anyway
Guarantee only side-effect-free degrades; gate writes separately.
User can't recoverMessage lacks next steps
Always end failures with what the user can do next.
Debugging tips
- Same failure recurs → classify beyond transient; add a specific rule.
- Degrades look like sloppy answers → label the degrade in the message.
- Aborts underreport → capture partial state and reasons in telemetry.
Interview checkpoint
Hands-on lab
- Record completed task ids as a checkpoint
- Simulate a crash mid-run
- Resume by skipping already-completed tasks
Checkpoint and Resume
25 min · intermediate
Quick quiz
Cap exhaustion should produce…
Divergence usually fixes itself via endless retries.