agent-loop
Designing agent retry logic that doesn't burn your token budget
Learn how to implement exponential backoff, retry budgets, and circuit breakers to build resilient AI agents that recover from failures without wasting tokens.
A retry that fires every second until an API responds can rack up thousands of dollars in token costs overnight. The right AI agent retry strategy balances resilience against waste: your agent should recover from transient failures without hammering endpoints or draining budgets. This article covers how to design retry logic that keeps agents reliable while protecting your token spend.
We'll walk through exponential backoff, retry budgets, structured error classification, and circuit breakers—each with worked examples and practical tradeoffs you can implement today.
Why AI agents need retry strategies in the first place
LLM API calls fail: rate limits, timeouts, temporary outages, and transient network errors all happen. An agent that handles a support ticket might call an external knowledge base, a Slack webhook, and GPT-4 in sequence—any of those can fail mid-flight. Without retries, one temporary glitch aborts the entire workflow. With naive retries (try again immediately, forever), you amplify the problem, burn tokens on repeated requests, and sometimes trigger cascading failures across the API provider.
A good retry strategy wraps each call in logic that distinguishes transient errors (worth retrying) from permanent ones (user permission denied, invalid syntax), waits progressively longer between attempts, and caps the total effort so a stuck agent doesn't loop until the credit card melts.
Classifying errors: retry vs. abort
Not every error deserves a retry. The first step is to categorize responses:
| Error type | Examples | Retry? |
|---------------------|---------------------------------------------------|--------|
| Transient/rate limit| HTTP 429, 503, timeout, network blip | Yes |
| Permanent client | HTTP 400, 401, 403, malformed JSON, invalid tool | No |
| Permanent server | HTTP 501 (not implemented) | No |
| Unclear | HTTP 500 without retry-after header | Maybe |
Inspect the HTTP status, error code, and any Retry-After header. OpenAI, Anthropic, and Azure OpenAI all return 429 for rate limits and often include a Retry-After hint. Respect that hint and add a small buffer—retrying sooner wastes tokens and risks further throttling.
Permanent errors should log, escalate to a human, or invoke a fallback path. Retrying a 403 Forbidden five times only delays the inevitable failure and inflates cost.
Exponential backoff with jitter
The classic pattern: wait a little, then longer, then longer still.
Example schedule:
Attempt 1: immediate
Attempt 2: wait 1 second
Attempt 3: wait 2 seconds
Attempt 4: wait 4 seconds
Attempt 5: wait 8 seconds (then stop)
With jitter (randomized delay), many agents hitting the same rate limit won't all retry at exactly the same moment:
const baseDelay = Math.pow(2, attemptNumber) * 1000; // milliseconds
const jitter = Math.random() * 0.3 * baseDelay;
const actualDelay = baseDelay + jitter;
Cap the maximum delay (for instance, at 16 seconds) to avoid unbounded waits. This structure spreads load and prevents thundering-herd retry storms.
Token budgets and retry ceilings
Every retry consumes tokens: the prompt is re-sent, the model generates a new response. A 2,000-token prompt retried five times costs 10,000 input tokens plus any output. Set a per-task token budget and track cumulative spend:
max_tokens_for_task = 15000
tokens_used = 0
for attempt in range(max_retries):
response = call_llm(prompt)
tokens_used += response.usage.total_tokens
if tokens_used > max_tokens_for_task:
log_and_escalate("Token budget exceeded")
break
if response.success:
return response
if not is_retryable(response.error):
break
wait_with_backoff(attempt)
This caps runaway cost even if the retry count is high. In Goatfied, each GOAT agent has a monthly budget, and retry logic respects that ceiling—work stops when the agent hits its limit, ensuring predictable spend.
Circuit breakers: stop retrying a broken service
If an external API is down, retrying every single agent task against it wastes tokens and slows recovery. A circuit breaker tracks recent failure rates and temporarily halts calls:
- Closed (normal): Calls go through; failures are counted.
- Open (tripped): All calls fail fast for a cooldown period (for example, 60 seconds).
- Half-open (testing): One probe call; if it succeeds, close the circuit; if it fails, reopen.
Example threshold: if five consecutive calls to external-kb.example.com fail, open the circuit for one minute. This prevents a hundred agents from each burning five retries against a service that's clearly offline.
Circuit breakers belong at the integration layer, shared across all agents, not per-task.
Structured logging and post-mortems
Every retry attempt should emit a structured log:
{
"agent_id": "support-goat-42",
"task_id": "ticket-9876",
"attempt": 3,
"error_type": "rate_limit",
"tokens_this_attempt": 2300,
"cumulative_tokens": 6900,
"backoff_ms": 4200
}
These logs feed dashboards that show retry frequency, token waste, and which endpoints flake most. Goatfied's Observability suite ingests OpenTelemetry traces and logs; you can alert when retry rates spike or when a single task burns through half its monthly budget in retries.
Post-incident, query logs to see whether a 10x token spike came from retries or from prompt drift, and tune budgets or backoff curves accordingly.
Fallback strategies beyond retries
Sometimes the best retry strategy is not to retry at all, but to try something different:
- Model fallback: If GPT-4 times out, fall back to a faster model (GPT-3.5) for non-critical steps.
- Cached responses: If an agent fetches the same knowledge-base article repeatedly, cache the response for a few minutes.
- Graceful degradation: Skip optional enrichment steps if they fail; deliver a partial result rather than aborting.
These patterns reduce retry load and keep agents moving when one dependency wobbles. In Goatfied's agent loop, coding agents test their own changes before opening a pull request; if the formatting tool times out, the agent can proceed with unformatted code and flag it for human review rather than retry indefinitely.
Putting it together: a checklist
- [ ] Classify errors; only retry transient failures (429, 503, timeouts).
- [ ] Implement exponential backoff with jitter and a maximum delay cap.
- [ ] Set a per-task token budget and stop retrying when it's exhausted.
- [ ] Deploy circuit breakers for shared external services.
- [ ] Emit structured logs with attempt count, error type, and cumulative token spend.
- [ ] Monitor retry rates and token waste; alert on anomalies.
- [ ] Consider fallback models, caching, and graceful degradation before adding more retries.
Frequently asked questions
How many retries should I allow before giving up?
Three to five attempts is common. Beyond that, the failure is likely not transient. Combine a retry ceiling with a token budget to prevent runaway cost.
Should I retry on HTTP 500 errors?
It depends. A 500 can be transient (temporary overload) or permanent (backend bug). If the API includes a Retry-After header or you know the service is flaky, retry with backoff. If 500 errors are rare, treat them as permanent after one or two attempts and escalate to a human.
How do I test retry logic without burning real API credits?
Mock the LLM client in your test suite to return failures on demand. Simulate sequences like [timeout, timeout, success] or [rate_limit, rate_limit, rate_limit] and verify your backoff curve and token accounting. Record real failure traces in staging, then replay them in CI.
Can retries ever make an outage worse?
Yes—if every agent retries aggressively at the same time, you amplify load on a struggling service (thundering herd). Jitter, circuit breakers, and respecting Retry-After headers prevent this. In a shared-tenant environment, coordinate retry logic at the platform level so one customer's retry storm doesn't degrade everyone else's experience.
How does Goatfied handle retries for its own agents?
Each GOAT operates within a monthly token budget and uses exponential backoff with jitter for transient API failures. Circuit breakers protect shared integrations, and the Observability suite logs every retry attempt with token counts so you can spot inefficiencies. Agents escalate to humans when retry budgets are exhausted or when errors are classified as permanent.