Skip to content
AI Learning Playground
LinkedInX
English
English

Failure Detection and Recovery

About 5 minutes

Shiori
Target audience: Those designing what happens when a workflow fails partway, those preventing silent failures
Prerequisites: Basic understanding of State and Handoff

In multi-step workflows, how mid-run failures are handled determines reliability. What goes undetected runs to completion on a wrong result.

Detection Comes From Completion Conditions

Detecting failure requires each step to have a defined completion condition. Without one, there is no way to judge whether the step finished.

How the failure appearsDetection method
The tool raises an exceptionDetected as an error
Processing succeeds but output misses requirementsRequires checking against the completion condition
Nothing happens and work continuesUndetectable without a completion condition
This table scrolls horizontally. Keyboard users can focus the table and use the left and right arrow keys.

The second and third are the dangerous ones. Because no error surfaces, only a mechanism catches them. Running verification right after a step with a hook is the reliable approach.

Choose the Fallback by Cause

Returning to the beginning every time is wasteful. Match the fallback point to the cause.

graph TD
    F["A step fails"] --> C{"Where is the cause"}
    C -->|Invalid input| Prev["Fix the previous step's output"]
    C -->|Wrong procedure| Same["Rerun the same step with changed conditions"]
    C -->|Assumption broke| Plan["Revisit the plan"]
    C -->|External factor| Wait["Retry after a delay"]
CauseFallback point
Faulty inputThe previous step
Problem in procedure or instructionThe same step, with changed conditions
Broken assumptionRevisit the plan
Transient external factorThe same step, after a delay
This table scrolls horizontally. Keyboard users can focus the table and use the left and right arrow keys.

Cap and Escalate

Automatic recovery needs a ceiling. Once the same step reaches it, escalate to a human rather than continuing.

Escalation should carry three things.

  1. Which step failed, and how many times
  2. The error, or the gap against the completion condition
  3. Which deliverables survive so far

Without these, the person starts by reconstructing the situation from scratch.

Do Not Discard Partial Results

Deliverables from completed steps should survive a failure. With progress held in state, successful steps never need redoing. Rolling back everything gets expensive in long workflows.

Summary

  • Failure detection requires completion conditions on each step
  • Failures without errors are the dangerous ones and need mechanical verification
  • Choose the fallback point by cause: previous step, same step, or the plan
  • Escalate past the retry ceiling, carrying the situation and surviving deliverables
Quiz