What 10,000 failed agent runs taught us about error recovery

Watching an AI agent fail is like watching a junior developer spiral: they make a confident edit, break the build, then frantically patch the error in a way that breaks something else. We've logged thousands of these failures across our agent loop, and the patterns are both humbling and instructive.

This isn't a post-mortem of catastrophic failures. Most agent errors are quiet: a missed constraint, a malformed import, a type error three files deep. But they compound. An agent that fails to recover cleanly from its first error will typically fail again within two subsequent edits. Understanding why—and how to interrupt the spiral—is the difference between an agent that ships working code and one that churns indefinitely.

The three failure modes that matter

After analyzing failure chains, we've clustered agent errors into three buckets that actually predict recovery difficulty:

Constraint violations happen when an agent ignores or forgets a boundary. It adds a dependency that's banned, writes to a read-only path, or introduces a pattern the team explicitly deprecated. These failures are clean: the constraint system catches them before execution, and the fix is usually surgical. The agent retries with updated context.

Compilation/validation failures surface after the edit but before runtime: TypeScript errors, lint violations, test failures. These are the sweet spot for recovery—if you catch them. An agent that sees Property 'userId' does not exist on type 'User' can usually backtrack and fix it. But if it doesn't get clear, actionable feedback, it guesses. That's when cascades start.

Runtime failures are the worst. The code compiles, passes static checks, and blows up in execution. The agent edited a function signature but missed a call site three modules away. Or it assumed a null-safe path that wasn't. Runtime failures require broader context to fix—often more than the agent originally had.

The critical insight: recovery success isn't about the error type, it's about the distance between the error and the agent's mental model. Constraint violations are close—the agent knew the rule existed. Runtime failures are far—the agent didn't know the dependency graph well enough to predict the break.

Why the second edit is the danger zone

We see a distinct spike in failure rate on the second edit after an error. The pattern:

1. Agent makes an initial change. Validation fails.

2. Agent receives error output, retries.

3. The retry introduces a new error—often in a different file or subsystem.

Why? Because the agent's second edit is contextually narrower. It's now focused on fixing the error message, not on the original task. It loses sight of the broader change it was making.

Here's a real example from a refactor task:

// Initial change: rename `getUser` to `fetchCurrentUser`

- export function getUser(id: string): User {

+ export function fetchCurrentUser(id: string): User {

Validation fails: Cannot find name 'getUser' in auth.service.ts. The agent retries:

// Second edit: fix the call site

- const user = getUser(session.id);

+ const user = fetchCurrentUser(session.id);

But it still doesn't see the other call site in permissions.ts, or the third in a test file. Each retry narrows focus. By the third edit, the agent is fixing errors, not completing the task.

Our fix: the validation loop must re-scope the agent's context after a failure. If a retry is needed, we re-run the plan phase with the error as additional input. The agent doesn't just patch—it re-plans with knowledge of what broke.

Small diffs are the recovery mechanism

Large diffs hide dependency ripples. When an agent makes a 200-line change across five files, a validation error often points to a symptom, not the root cause. The agent can't easily isolate which of its fifteen edits triggered the break.

We learned this by comparing recovery rates across diff sizes. Edits under 50 lines had a recovery success rate we'd estimate in the 70–80% range. Edits over 150 lines? Recovery became a coin flip, often requiring human intervention.

The solution isn't to ban large changes—it's to decompose them. If a task requires touching ten files, the agent plans it as a sequence of smaller, validated steps:

1. Update the core function signature.

2. Validate. If it fails, fix before proceeding.

3. Update immediate call sites.

4. Validate again.

5. Update downstream dependencies.

Each step is a commit point. If step 3 fails, the agent has a clean rollback boundary. It doesn't need to unwind the entire change—just the last delta.

This is why the Goatfied agent loop enforces plan → constrain → edit → validate → retry as discrete phases. The validate gate isn't optional. The agent cannot proceed to the next logical edit until the current one passes compilation, linting, and any defined tests.

Error messages are training data for the next loop

Most agent frameworks treat error output as ephemeral: log it, maybe show it to the user, move on. We treat it as context for the next iteration.

When an agent hits a TypeScript error, we don't just pass back the raw compiler output. We parse it, extract the file location and error code, and inject it into the agent's next planning step as a constraint:

> "Your last edit introduced TS2339 in src/auth.service.ts:42. The property userId does not exist on the resolved type. Re-plan your approach to avoid this error."

This sounds obvious, but many agent systems skip the re-plan. They jump straight to a fix attempt, which is how you get thrashing.

The same applies to runtime errors. If a test fails, we capture the stack trace and the assertion message, then ask the agent: "Given this failure, what should have been different about your edit?" This forces the agent to reason backward from the error to the change, which is a higher-leverage mental model than "fix the test."

Audit trails make recovery debuggable

When an agent fails three times and finally succeeds on the fourth try, the victory is hollow if you can't explain why the fourth attempt worked. We've started treating the retry loop itself as an auditable artifact.

Every run produces a structured log:

• Plan phase: what the agent intended to do.
• Constrain phase: which rules applied.
• Edit phase: the diff it generated.
• Validate phase: which checks ran, which failed.
• Retry phase: how the agent revised its approach.

If the agent eventually succeeds, we have a clear chain showing what it learned. If it fails after N retries, we have evidence of why—usually a constraint it couldn't satisfy or context it never acquired.

This matters for two reasons. First, it lets engineers tune the loop. If agents consistently fail on a particular rule, maybe the rule is too rigid or poorly expressed. Second, it builds trust. Teams tolerate agent failures better when they can see the reasoning, not just the outcome.

What we still get wrong

Recovery from cascading failures remains hard. An agent edits a file, breaks a build, fixes the build but breaks a test, fixes the test but reintroduces the original error. We don't have a clean way to detect these cycles early. Right now, we set a retry limit (usually three attempts) and escalate to human review if the agent can't converge.

We're also still learning how to handle ambiguous validation failures. A flaky test fails intermittently—does the agent retry, or does it assume the test is the problem? A linter flags a style violation that contradicts a recent commit—does the agent follow the rule or match the existing code? These edge cases require judgment we haven't fully encoded.

The takeaway for agent builders

If you're building or integrating AI agents into a development workflow, here's what the data suggests:

• Make validation mandatory and fast. Every edit should pass through compile/lint/test gates before the next edit begins.
• Keep diffs small and scoped. Agents are better at recovering from narrow, isolated failures than broad, tangled ones.
• Treat errors as input to re-planning, not just fix prompts. The agent should revise its strategy, not just its code.
• Log the full loop, not just the final output. You'll need it to debug, tune, and explain.

Agent failures aren't a sign the technology doesn't work—they're a sign you're using it on real problems. The goal isn't zero failures; it's fast, transparent recovery.

Related reading

• The Goatfied agent loop: how we ship code that actually compiles first try
• Why AI agents fail on the second edit, and how validation loops fix it