Making Agents Reliable: Retries, Timeouts, Checkpoints

Updated Nov 16, 2026 · 4 min read
On this page

    An agent works perfectly in the demo: five steps, every one succeeds. In production, tasks take 20 steps, tools time out, the provider returns 429s at peak hours, and the model occasionally passes a bad argument. Each problem is rare, yet a third of runs fail. Reliability for agents is mostly about arithmetic and engineering, not about a smarter model.

    How it works

    Errors compound

    If every step succeeds 95% of the time, a 20-step run succeeds only about 36% of the time.
    If every step succeeds 95% of the time, a 20-step run succeeds only about 36% of the time.

    A run succeeds only if every step does, so per-step success multiplies: 0.95²⁰ ≈ 0.36. Two levers follow: make each step more reliable and make the run recover from failed steps instead of failing as a whole. Shorter runs (fewer steps) help too.

    Guard every step

    The same few guards apply to model calls and tool calls alike.
    The same few guards apply to model calls and tool calls alike.
    • Timeouts on every model and tool call, so one hung call doesn't hang the run.
    • Retries with exponential backoff and jitter for transient errors only (timeouts, 429, 503), with a cap.
    • Validation of model output (schema, allowed values, business rules) and of tool results before using them.
    • Fallbacks: a second model or provider when the primary is down or rate-limited.
    • Budgets: maximum steps, wall-clock time, tokens and cost per run.
    import random, time
    
    class TransientError(Exception):
        """Timeouts, 429s, 503s: worth retrying."""
    
    def call_with_retry(fn, max_attempts=4, base=0.5, cap=8.0, sleep=time.sleep):
        for attempt in range(1, max_attempts + 1):
            try:
                return fn()
            except TransientError:
                if attempt == max_attempts:
                    raise
                delay = min(cap, base * 2 ** (attempt - 1))
                sleep(random.uniform(0, delay))          # "full jitter" spreads retries out

    Classify failures

    Retrying is right for some failures and harmful for others.
    Retrying is right for some failures and harmful for others.
    • Transient (timeouts, rate limits): retry with backoff.
    • Permanent (not found, permission denied): don't retry; return a clear error to the model so it can change approach or ask the user.
    • Model mistakes (invalid arguments, wrong tool, repeating itself): validate, send specific feedback, cap the number of corrections, detect loops (same call with the same arguments several times).
    • Stuck or risky (no progress, or about to do something high-impact): checkpoint and escalate to a person.

    Recover instead of restarting

    • Checkpoint after each step so a crash resumes where it stopped (see Agent Memory and State).
    • Idempotency keys on side-effecting tools, so retries and resumes never double-charge or double-send.
    • Verification steps at milestones ("did the file actually change? do the tests pass?") catch silent failures early.
    • Graceful degradation: return partial results with a clear note rather than failing silently.

    Trade-offs and costs

    • Retries add latency and cost; aggressive retries amplify outages (retry storms). Cap them and respect Retry-After.
    • Validation and verification steps cost extra calls but are far cheaper than a failed run.
    • Human escalation improves outcomes but needs a UI, a queue and people.
    • Determinism: lower temperature for tool-calling steps reduces random mistakes, at little cost.

    In production

    • Measure per-step success rates by tool and by model, not just run success, to find the weakest link.
    • Alert on rising retry rates, loop detections, budget hits and escalation rates.
    • Chaos testing: inject tool timeouts and errors in evals to see whether the agent recovers.
    • Failure modes: retry storms during a provider outage; non-idempotent tools repeated after resume; infinite correction loops; agents reporting success after a failed tool call.

    When to reach for it

    As soon as an agent runs more than a few steps, touches external systems, or runs unattended. The longer and more autonomous the run, the more these guards matter.

    Example

    A data-migration agent had 64% run success on 50 eval tasks averaging 18 steps. Traces showed most failures were transient API errors and two kinds of bad tool arguments. Fixes: retries with jitter on transient errors, schema validation with a one-shot correction message, a loop detector, checkpoints after each table, and a verification query after each migration step. Run success rose to 94%, with average cost up only 8% (example numbers).

    Common mistakes

    • Retrying everything, including permanent errors.
    • Retries without jitter or caps, causing retry storms.
    • No step or cost budget.
    • Trusting the model's claim of success instead of verifying state.
    • Non-idempotent side effects in a system that retries.

    Say this in the interview

    "Per-step errors compound, so a 95%-reliable step gives about 36% success over 20 steps. I'd raise per-step reliability with timeouts, retries with backoff and jitter for transient errors, output validation and fallback models; and make runs recoverable with checkpoints, idempotent tools, verification at milestones and escalation when stuck. Failures are classified: retry transient ones, return permanent ones to the model, correct model mistakes a bounded number of times."

    Interview questions on this

    1. Why do long agent runs fail more? Success probability multiplies across steps.
    2. Which errors should you retry? Transient ones (timeouts, 429, 503), with backoff, jitter and a cap.
    3. How do you stop an agent from repeating a side effect after a retry? Idempotency keys checked by the tool.
    4. How do you detect a stuck agent? Repeated identical calls, no state change across steps, or budget nearly used without progress.
    5. How do you test reliability? Inject tool failures and latency in evals and measure recovery rate.

    Used in

    • AI Engineering: Tool Calling and the Agent Loop; Agent Memory and State; Evaluating Agents; Your Agent Is Stuck in a Loop (practical question); Model Routing and Fallbacks.
    • AI System Design: Design Operator; Design Deep Research.