Debugging Ai Generated Code With Checklists Failure Patterns: practical systems guide

AI-generated code breaks in predictable ways. The agent proposes a fix that compiles but fails at runtime. It misreads the schema and generates a malformed query. It refactors a method signature but misses one callsite buried three levels deep. If you're relying on an LLM to draft pull requests or close tickets, you need a systematic approach to catch these failures before they merge—ideally without burning engineer hours on manual review.

The most effective pattern we've seen is a **failure-mode checklist**: a short, ordered list of common AI code defects that you check in CI or during human review. This isn't about perfection. It's about stopping the 80% of avoidable breakage that follows repeatable patterns. Below is a practical system for building and using these checklists, grounded in patterns we've observed running automated tasks through Goatfied's agent loop (plan → constrain → edit → validate → retry).

Why checklists beat ad-hoc review

When a junior engineer reviews AI-generated code, they often scan for syntax correctness and aesthetic issues—indentation, naming, obvious typos. The subtler bugs slip through: a changed function signature with orphaned callers, a missing null check in a newly added branch, a dependency version bump that breaks a transitive import.

A checklist forces you to look for specific failure modes in a defined order. It's fast (each item takes 10–30 seconds), repeatable (junior and senior engineers run the same checks), and improvable (you add items as new failure patterns emerge). Think of it as a lightweight pre-merge gate that complements your existing compile/lint/test pipeline.

Core failure patterns to check

Start with these six categories. They cover the majority of silent breakage we've debugged in AI-drafted PRs:

1. Signature drift and orphaned callers

The agent renames a function parameter or changes a return type but doesn't propagate the change to all call sites. Compiled languages catch some of this; dynamic languages and cross-module calls often don't.

**Check:** Search the codebase for the old function name or parameter. If you changed `processOrder(orderId)` to `processOrder(request)`, grep for `processOrder(` and verify every match now uses the new signature.

**Example CI rule:**

# After AI edit, diff function signatures and grep for old patterns

git diff main --stat | grep -E '\.(ts|py)$' | xargs grep -n 'oldFunctionName'

2. Missing null/undefined guards in new branches

AI often adds a new conditional path—`if (config.featureFlag)`—without checking whether `config` itself can be null. The happy-path test passes; production crashes when `config` is undefined.

**Check:** For every new `if` or `switch` statement, trace back to the source of the tested variable. Does the schema or type definition allow null? If so, does the new code handle it?

**Manual heuristic:** Scan new conditionals for object property access (`obj.prop`) without a prior null check on `obj`.

3. Schema / API contract mismatches

The agent drafts a database query or API call based on outdated documentation or an incorrect assumption. The code compiles, the test mocks return happy data, but the real endpoint returns 400 or the query returns empty.

**Check:** Cross-reference new query/API code against the actual schema or OpenAPI spec. If the AI added a field `user.email`, confirm that field exists in your current schema version.

**Tool-assisted:** Run a local schema validator or contract test. In Goatfied's workflow, the validate step can invoke a lint rule that checks Prisma queries against `schema.prisma` or API calls against a local OpenAPI file.

4. Dependency version conflicts

The agent bumps a library version to unlock a new feature but doesn't check transitive dependencies. Your build passes in CI (which installs fresh), but developers see peer dependency warnings or runtime import errors.

**Check:** After any `package.json` or `requirements.txt` change, run `npm ls` / `pip check` and scan for warnings. Better: lock-file diff review—look for unexpected major-version bumps in transitive deps.

**Goatfied approach:** The constrain step can enforce a rule: "Only patch/minor bumps allowed unless explicitly requested." The agent produces the edit, the validate gate runs `npm ci && npm audit`, and the retry loop fixes conflicts before human review.

5. Silent test coverage gaps

The AI writes or modifies a function but doesn't add a test for the new error path. Existing tests pass (they cover the old logic), so the PR looks green. The untested branch fails in production.

**Check:** Diff test files against code files. If you added 20 lines to `billing.ts`, did you add or modify a test in `billing.test.ts`? Use coverage diffing tools like `jest --coverage --changedSince=main` to surface new uncovered lines.

**Rule of thumb:** Any new `throw`, `return null`, or `else` branch should have at least one test line exercising it.

6. Copy-paste drift in similar functions

The agent duplicates a block of code (e.g., authentication logic) across two handlers. It tweaks one copy but not the other, creating subtle inconsistencies—one handler logs errors, the other swallows them.

**Check:** Run a duplicate-code detector (`jscpd`, `pylint --disable=all --enable=duplicate-code`) on the diff. Flag blocks >10 lines that appear twice with minor differences.

**Resolution:** Either extract a shared helper or document why the duplication is intentional (and test both paths independently).

Structuring the checklist for CI and human review

Break the checklist into two stages:

**Automated (CI gate):**

• Compile, lint, type-check
• Test suite (including coverage diff)
• Schema/contract validation
• Dependency conflict check (`npm ls`, `pip check`)
• Duplicate-code scan

**Human spot-check (2–5 min):**

• Signature drift: grep for old function names
• New conditionals: null/undefined guards
• Copy-paste logic: duplication report review
• Test coverage: any new error paths untested?

In Goatfied, this maps cleanly to the validate step (automated gates) and a lightweight human review before merge. The agent retries if validation fails, so most PRs that reach human review have already passed the mechanical checks.

Growing the checklist over time

When a bug escapes to production, run a quick postmortem: which checklist item would have caught it? If none, add a new item. Examples we've added after real incidents:

• "Verify new environment variables are documented in `.env.example`"
• "Check that new database migrations include a rollback script"
• "Scan for hardcoded `localhost` URLs in new integration code"

Keep the list short—10 items max. If it grows beyond that, automate the common ones and keep only the judgment-heavy checks for humans.

Practical tradeoffs

Checklists add friction. They slow down fast iteration, especially on prototypes where breakage is acceptable. Use them selectively:

• **Enable always:** Production PRs, customer-facing features, schema changes.
• **Disable or lighten:** Internal tools, proof-of-concept branches, hotfixes under time pressure (but add a follow-up ticket).

The goal isn't zero defects. It's to catch the predictable, boring failures before they consume engineer time in production debugging or rollback meetings.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [Benchmarks playbook #8: practical Goatfied tactics for shipping PRs](/blog/goatfied-benchmarks-playbook-8)