benchmarks
We pointed six AI coding tools at a 400k-line monorepo. Here's what broke.
We tested six AI coding assistants on a 400,000-line Python monorepo with real feature tasks to document how they handle cross-service changes at scale.
We took GitHub Copilot, Cursor, Cody, Amazon Q Developer, Windsurf, and Goatfied into a Python monorepo with 400,000 lines across twelve services and asked each to implement the same eight feature tasks. Not toy examples—real tickets pulled from our backlog involving cross-service schema changes, rate-limiting middleware, and a background job refactor. The goal wasn't a beauty contest; we wanted to see where conversation-driven coding breaks down when repository scale pushes against context windows, LSP overhead, and the sheer cognitive load of multiple interdependent modules.
None of the tools sailed through unscathed. Some fell over on first contact with the codebase. Others generated plausible-looking edits that compiled but silently broke invariants three modules away. A few succeeded on isolated tasks but couldn't maintain consistency across the multi-file changes that typify monorepo work. What follows is an honest tally of failure modes, workarounds, and the rare pleasant surprises.
The test environment
The monorepo is a typical early-scale setup: twelve Python services under a single Git root, shared libraries in libs/, centralized CI running Ruff, mypy strict mode, pytest with ~3,200 tests. Services range from 8k to 90k lines; average module depth is four levels. We used Docker Compose for local dev and Kubernetes manifests in deploy/. Total repository size on disk: 1.2 GB including .git, 180 MB of checked-in code.
Each tool saw the same machine: 32 GB RAM, eight-core i7, Ubuntu 22.04. We gave every agent the same prompt template for each task, waited up to ten minutes per attempt, and recorded whether the result passed lint, type-check, and tests without manual fixups. We did not cherry-pick runs. If an agent asked clarifying questions, we answered them once; if it looped or stalled, we noted the point of failure.
Task 1: Add a rate-limiting decorator to the API gateway
This should have been straightforward—introduce a Redis-backed rate limiter in services/gateway/middleware/ratelimit.py, apply it to three endpoints, add configuration in gateway/config.py, and write two unit tests. Total diff: ~120 lines across four files.
Copilot and Cody both completed the decorator but hard-coded the Redis host instead of reading from the config object. Tests passed in isolation but would fail in staging. Copilot also imported time.sleep instead of using an async-compatible sleep, which broke the event loop under load.
Cursor nailed the implementation but placed the new module in libs/common instead of services/gateway/middleware, violating the repo's service-isolation rule. When we pointed this out in chat, it moved the file but left the old import paths intact in the three call sites. Lint failed.
Amazon Q generated a correct decorator but suggested installing redis-py-cluster, a library we don't use, and added it to a nonexistent requirements.txt instead of the Poetry pyproject.toml. It also wrote synchronous Redis calls in an async endpoint, causing runtime blocking.
Windsurf applied the decorator correctly, updated config, wrote tests—then inexplicably reformatted 200 unrelated lines in gateway/app.py to change quote style, triggering a CI noise explosion.
Goatfied succeeded after one retry. The initial plan proposed putting configuration in a new settings.toml, which didn't match our existing config.py pattern. We rejected that plan step; the agent regenerated with the correct config merge and passed lint + type-check + tests on the second attempt.
Task 2: Refactor shared pagination logic into libs/common
Three services had near-identical paginate() functions. The task: extract one canonical version into libs/common/pagination.py, add type stubs, update the three call sites, ensure tests still pass.
GitHub Copilot created the shared function but retained positional-only arguments in the signature, while two of the three services used keyword arguments. Tests failed with TypeError. It did not catch this because it ran no validation between editing and declaring success.
Cody extracted the function and updated imports, but missed one call site in services/orders/api/handlers.py buried four directories deep. The build passed (that handler wasn't covered by existing tests), but manual testing revealed a 500 error on the orders list endpoint.
Cursor completed the refactor cleanly and even added a docstring with usage examples. No complaints.
Amazon Q wrote the shared function, then got confused about the module path and tried to import from common.pagination instead of from libs.common.pagination. The agent looped three times trying to fix the import, eventually timing out.
Windsurf succeeded but added an unnecessary @functools.lru_cache decorator "for performance," which broke pagination for mutable query objects. The decorator was never mentioned in the task; the agent invented an optimization that introduced a bug.
Goatfied proposed the refactor, showed a diff of all three call sites in the plan, passed constraint checks (which flagged one call site using a deprecated kwarg), and completed the task in one pass.
Task 3: Add a new status enum to the orders schema and propagate it
A simple schema change: add OrderStatus.PENDING_REVIEW to libs/models/order.py, create an Alembic migration, update the serializer in services/orders, and handle the new state in the fulfillment worker in services/fulfillment.
Copilot added the enum variant and serializer change but forgot the migration entirely. Manual alembic revision was required. It also didn't touch the fulfillment worker, leaving dead states in production.
Cody generated the migration but used a raw SQL ALTER TYPE statement that would fail on Postgres <12. The agent had no awareness of our DB version constraints.
Cursor created the enum, migration, serializer, and fulfillment handler. Clean run. The migration even included a comment explaining the enum addition.
Amazon Q added the enum but then regenerated the entire Alembic migration history, suggesting we re-run all migrations from scratch. Catastrophic for a live system.
Windsurf succeeded on the enum and serializer, wrote the migration, but in the fulfillment worker it added a match statement (Python 3.10 syntax) while our deployment runs 3.9. CI failed on syntax error.
Goatfied proposed the four-file change, constraint validation caught the missing migration during the plan stage, agent added it, and the edit passed. The diff was minimal and auditable.
Where context windows collapse
By task four—a cross-service change touching six files across three services—most agents hit wall. GitHub Copilot lost track of which service owned which model. Cody repeatedly re-read the same files, burning time in LSP indexing. Amazon Q hallucinated a services/shared directory that doesn't exist. Windsurf edited the wrong config.py (we have four).
Cursor and Goatfied fared better, but for different reasons. Cursor uses a smarter file-selection heuristic and caught most of the relevant modules. Goatfied's planning phase explicitly lists files before editing, so context misses surface early. You reject a bad plan instead of discovering broken imports ten minutes later.
The testing blindspot
Only Goatfied and Cursor ran validation automatically after generating code. The others declared success without executing pytest. This led to silent breakage: an agent would close the task, you'd run tests manually, and three modules would fail. In a monorepo, this compounds fast—one bad import can cascade into thirty test failures.
Tools that run tests in the loop caught their own mistakes and retried. Tools that don't left cleanup to the human.
The diff readability problem
Even when agents succeeded, PR review was painful. Windsurf and Amazon Q often reformatted entire files when asked to change one function, mixing semantic edits with whitespace churn. Cursor was better but occasionally moved imports or reordered class methods for no stated reason.
Goatfied's diffs were consistently small and single-purpose because the constraint phase rejects unrelated changes before they enter the edit. This mattered more than we expected—when reviewing a 12-file PR, you need clean diffs or you miss the bug hiding in line 47.
Takeaways for monorepo workflows
If your repository is small (under 50k lines, five services or fewer), most conversation-driven agents work fine. Scale past that and you need tooling designed for multi-module coherence: explicit plans, compile gates, and automatic rollback when validation fails.
The agents that struggled shared three traits: no pre-edit planning, no mandatory validation step, and no scoping mechanism to prevent unrelated changes. The agents that succeeded treated code generation as a constrained multi-step process, not a single LLM call.
Monorepos punish tools that conflate writing code with changing code. Writing is generative; changing is surgical. The latter requires guardrails.