Debugging Ai Generated Code With Checklists Design Tradeoffs: practical systems guide

# Debugging AI-Generated Code With Checklists: Design Tradeoffs

You push a fix that looked clean in the diff, merge it, and thirty minutes later someone pings: "The migration fails in staging because the script assumes UTC timestamps but our test DB uses local time." The AI agent got the core logic right but missed a boundary condition you would have caught in code review. This pattern—correct on first inspection, wrong under real conditions—is the central debugging challenge with AI-generated code.

Checklists feel like an obvious answer. Run every diff through a gate that checks for null handling, error paths, timezone assumptions, lock contention. But practical checklist design forces you to pick between catching more edge cases and slowing down every change. This guide walks through that tradeoff and shows how to build checklists that actually work in an agent loop.

Why checklists fail when you copy-paste from code review guides

Most engineering teams have accumulated code review wisdom: check for SQL injection, confirm error messages don't leak stack traces, verify new endpoints have auth middleware. The mistake is turning these into a literal checklist for the AI to validate against.

The problem is context collapse. A human reviewer sees "check for SQL injection" and immediately knows whether the diff touches database queries. An LLM given the same instruction will either hallucinate a SQL risk in a pure data transformation function or confidently report "no SQL found" when the code uses an ORM that obscures the query construction.

Effective checklists for AI-generated code must be both **mechanically verifiable** where possible and **narrowly scoped** to what the current diff actually changes. If your checklist asks "Does this handle null inputs correctly?" across an entire file, you're asking the agent to audit legacy code it didn't write. Scope the question: "For each function modified in this diff, are the newly added calls defensive against null arguments from the calling context?"

The compile-first gate as your baseline checklist

Before you write custom validation rules, make sure you're actually running the compile/lint/test suite as a hard gate. This sounds trivial but most AI coding tools treat test failures as suggestions.

In Goatfied's agent loop, the validate step is blocking: if `cargo build` or `npm run typecheck` fails, the diff doesn't proceed to retry—it gets constrained by the error and re-edited. This catches entire categories of bugs (type mismatches, import errors, unused variables) without custom rules.

# Example: validation gate in .goatfied/config.yml

validate:

  build: "cargo build --all-features"

  test: "cargo test --lib"

  lint: "cargo clippy -- -D warnings"

  # If any fail, agent re-enters edit phase with errors as constraints

Your checklist should start here and only add rules for issues that slip through static analysis. If your linter already flags unwrapped `Result` types in Rust, don't write a separate checklist item for error handling.

Partitioning checklists by change type

A diff that adds a new API endpoint needs different scrutiny than a refactor that extracts a helper function. Applying every rule to every diff creates false positives and slows the loop.

Partition your checklist by **change classifier**. Before validation, tag the diff:

• **New public surface** (API route, exported function, CLI command): Check auth, rate limiting, input validation, error response format
• **Database migration**: Check reversibility, index strategy, null handling in existing rows
• **Async boundary** (new goroutine, tokio spawn, JS Promise): Check cancellation, error propagation across the boundary, resource cleanup
• **Dependency update**: Check for breaking changes in CHANGELOG, verify tests cover the integrated behavior

The agent can self-classify with a prompt like "Does this diff add a new HTTP endpoint?" but for reliability, use mechanical signals: does the diff touch `routes.rs` or add a `@app.route` decorator? Does it create a new `.sql` file in `migrations/`? Mechanical checks avoid the agent misclassifying its own work.

Trading off depth for iteration speed

Every additional checklist item adds latency to the plan → edit → validate loop. If validation takes thirty seconds because you're running five static analyzers and a custom script that checks for hardcoded secrets, you slow the agent's ability to iterate on the fix when validation fails.

The tradeoff: **shallow checklists with fast feedback** let the agent try more solutions. **Deep checklists** catch more on the first attempt but penalize retries.

In practice, tiered validation works best:

1. **Fast gate** (< 5 seconds): Compile, type-check, fast unit tests, basic linting

2. **Standard gate** (< 30 seconds): Full test suite, integration tests for touched modules

3. **Pre-merge gate** (< 3 minutes): E2E tests, security scans, deployment dry-run

The agent loop runs tiers 1 and 2 on every retry. Tier 3 runs once before the PR is marked ready. This keeps the edit-validate cycle tight while still catching expensive-to-fix issues before merge.

Checklist items that actually catch AI-specific bugs

Beyond standard code quality rules, certain checks target failure modes specific to LLM-generated code:

**Boundary condition coverage**: Did the agent add a function that handles a list but never tested with an empty list? Check for test cases covering `len(input) == 0`, `input == None`, single-element inputs.

**Context leakage**: Does the new code reference a variable or import that only exists in the agent's *plan* but wasn't actually added to the file? This happens when the agent describes what it will do, then forgets to do it.

**Copy-paste inconsistency**: Did the agent duplicate a block to handle two cases but only update the variable names in one? Check for near-identical blocks with suspicious name reuse.

**Overfitting to the prompt**: If you asked the agent to "add caching to speed up repeated queries," did it cache unconditionally, including mutations? Check that cache keys and invalidation logic match the operation semantics.

You can encode some of these as linter plugins (e.g., a rule that flags functions with no test coverage for empty-collection inputs) but some require agent self-review prompts: "For each conditional branch in your diff, confirm at least one test exercises that branch."

When to let the human decide

Some checklist items can't be mechanically verified and shouldn't be auto-passed by the agent. These need a human gate:

• Does this change preserve backward compatibility with the deployed API version?
• Is the new feature flag name consistent with our existing naming scheme?
• Does the error message expose information useful for debugging without leaking internal state?

Mark these as **manual review required** in your workflow config. The agent completes its loop, but the PR doesn't auto-merge. A staff engineer confirms the subjective parts before deploy.

This is especially important for diffs that touch authentication, billing logic, or data retention policies—domains where a missed edge case has disproportionate impact.

Checklists as diff-scoped constraints, not repo-wide audits

The biggest design mistake: asking the agent to validate the entire codebase state rather than just its changes. "Does the application handle rate limiting correctly?" is an audit. "Does the new endpoint in this diff apply the standard rate-limit decorator?" is a scoped check.

Goatfied's constraint system helps here: when validation fails, the error becomes a constraint on the next edit attempt. If your checklist item is "confirm error paths return structured JSON," and the diff returns plain text, the next edit gets that failure injected as a requirement. The agent doesn't re-audit—it fixes the specific violation.

Scoped checklists make the agent loop converge faster because retries have a clear target, not an open-ended "improve code quality" prompt.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [Benchmarks playbook #8: practical Goatfied tactics for shipping PRs](/blog/goatfied-benchmarks-playbook-8)