A reproducible methodology for benchmarking multi-file refactors

The hardest part of evaluating AI coding tools isn't running them—it's proving the results mean anything. Most teams evaluate refactoring assistants by pointing them at a handful of tasks, eyeballing the diffs, and declaring a winner. This works until you realize the sample was cherry-picked, the prompts were inconsistent, or the "passing" code doesn't actually compile in a fresh checkout.

A reproducible methodology solves this by making every step of the evaluation auditable, repeatable, and falsifiable. Below is the approach we've refined at Goatfied after running hundreds of multi-file refactor benchmarks across real-world repositories—designed to withstand scrutiny from skeptical teammates and to catch the subtle failures that manual review misses.

Define pass criteria before you run anything

Start by writing down what "correct" means for each task. Not vibes—concrete gates. For a refactor that extracts a utility function and updates call sites, your criteria might be:

• The codebase builds without errors (tsc --noEmit or cargo check).
• All existing tests pass.
• The new utility function is imported and called correctly in at least N files.
• No commented-out code or leftover placeholders like // TODO: implement.

Write these as executable checks. A shell script that exits non-zero on failure is enough. The goal is to eliminate "it looks good to me" as a judgment.

If you're benchmarking a large-scale rename or API migration, add a semantic test: does a representative end-to-end scenario still work? This catches the cases where the code compiles but the logic is subtly broken.

Use clean, hermetic snapshots

Every benchmark run must start from an identical repository state. Uncommitted changes, stale node_modules, or cached build artifacts will poison your results.

Recommended pattern:

#!/usr/bin/env bash

set -euo pipefail



SNAPSHOT_REF="main"  # or a specific commit SHA

WORK_DIR=$(mktemp -d)



git clone --branch "$SNAPSHOT_REF" <repo-url> "$WORK_DIR"

cd "$WORK_DIR"



# Install dependencies with locked versions

npm ci  # or pip install -r requirements.txt, etc.



# Run the tool under test

<tool-specific invocation>



# Run validation

./scripts/validate.sh

The key is that anyone else can run this script six months from now and get the same initial conditions. If your tool requires local configuration files, check them into a benchmarks/ directory in version control.

Isolate the prompt or instruction set

The biggest source of irreproducibility in coding tool benchmarks is inconsistent human input. If you're testing GitHub Copilot Chat against Cursor against an agentic system, you need to give each tool semantically equivalent instructions—but phrased in the way that tool expects.

For Copilot Chat, that might be a natural-language message in the sidebar. For Cursor, it could be a composer prompt. For Goatfied, you'd write a plan step in the agentic loop and let the system decompose it into constrained edits.

The trick is to version-control your prompts as structured data:

{

  "task_id": "extract-validation-util",

  "instruction": "Extract the email validation logic in user.ts and auth.ts into a shared validateEmail() function in utils/validation.ts. Update all call sites to import and use the new function.",

  "expected_files_changed": ["user.ts", "auth.ts", "utils/validation.ts"],

  "timeout_seconds": 300

}

Run each tool with its native interface, but derive the input from the same canonical instruction. Document any adaptations you make ("Cursor requires explicit file paths, so we added them to the prompt").

Measure what matters: compile gates and test gates

The two non-negotiable checks are:

1. Does the code build? Run the project's type checker, linter, or compiler. Zero warnings is the bar for production-grade refactors. If your tool produces a diff that introduces a TypeScript error three files away from the edit site, that's a failure—even if the immediate change looks correct.

2. Do the tests pass? Run the full suite, not just unit tests. Integration and end-to-end tests catch the semantic breaks that type systems miss.

For languages without compile-time checks (Python, Ruby), add a smoke test that imports the modified modules and exercises key code paths. A refactor that passes pylint but crashes at runtime is still broken.

Track these as binary outcomes (pass/fail) and as a percentage of tasks where the tool required zero manual fixes to reach a passing state. That percentage is your headline metric.

Count the number of reversible attempts

Agentic tools like Goatfied retry failed edits automatically. Traditional assistants like Copilot typically give you one shot, and you manually iterate.

Log how many attempts each tool needed to reach a passing state:

• Attempt 1: tool generates initial diff
• Validation fails (e.g., type error in auth.ts)
• Attempt 2: tool regenerates or adjusts
• Validation passes

This separates tools that get it right the first time from those that rely on a feedback loop. Neither is inherently better, but the distinction matters for workflows where human review happens between attempts vs. fully automated agent loops.

Goatfied's architecture makes this explicit: the plan -> constrain -> edit -> validate -> retry cycle is instrumented, so you can measure how many validate/retry steps were needed. For tools without native retry, you can simulate it by feeding compiler errors back as additional context and counting iterations manually.

Track files touched vs. files required

A refactor that changes 47 files when only 8 needed modification is a red flag. Either the tool misunderstood the scope, or it's making overly broad edits.

Diff the tool's output against your expected file list:

expected = {"user.ts", "auth.ts", "utils/validation.ts"}

actual = get_changed_files_from_diff()



false_positives = actual - expected

false_negatives = expected - actual

False positives suggest the tool is over-editing. False negatives mean it missed required changes. Both tank the reproducibility of the refactor because a human has to step in and fix the scope.

Run the same benchmark across tool versions

Tools improve. To avoid "we benchmarked the old version" accusations, pin tool versions and re-run periodically.

For VS Code extensions, that means installing specific .vsix files. For API-based tools, log the model version (e.g., gpt-4-turbo-2024-04-09). For Goatfied, you can pin the agent runtime version in your self-hosted deployment config.

Create a matrix:

| Tool | Version | Task ID | Compile | Tests | Attempts | Files |

|---------------|---------------|---------|---------|-------|----------|-------|

| Cursor | 0.41.0 | task-01 | ✓ | ✓ | 1 | 3 |

| Goatfied | 2024-12-01 | task-01 | ✓ | ✓ | 2 | 3 |

| Copilot Chat | 0.22.1 | task-01 | ✗ | n/a | 1 | 5 |

Re-running this quarterly shows whether improvements in the underlying models or tool logic translate to better real-world outcomes.

Publish the full dataset

Reproducibility dies in private spreadsheets. Publish your task definitions, validation scripts, and raw results in a public repository. Redact proprietary code if needed, but share enough that someone else can replicate your setup.

At minimum:

• tasks/ directory with one JSON file per task
• scripts/validate.sh showing the compile/test commands
• results.csv with pass/fail outcomes and metadata
• README.md explaining how to run the benchmark end-to-end

This isn't just for transparency—it lets other teams extend your work. If someone finds a prompt phrasing that improves Cursor's performance on your tasks, they can submit a PR and you can re-run to confirm.

Handle edge cases explicitly

Real-world refactors hit edge cases: generated code, vendored dependencies, files that shouldn't be edited. Define how your benchmark handles these upfront.

For example, if a task involves files in node_modules/, do you let the tool edit them (almost always wrong) or do you fail the run immediately? If the tool produces a diff with merge conflict markers, is that an automatic failure or do you attempt to resolve it programmatically?

Document these decisions so future benchmark runs stay consistent.

Related reading

• Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks
• We pointed six AI coding tools at a 400k-line monorepo. Here's what broke.