Context Window Budgeting Under Token Pressure Design Tradeoffs: practical systems guide

# Context Window Budgeting Under Token Pressure: Design Tradeoffs

You have 128k tokens available and a monorepo with 2,400 TypeScript files. The agent needs to refactor a cross-cutting interface used in 47 modules. Do you send the full call graph? Summaries of each file? Just the interface definition and hope the LLM infers correctly?

This isn't a hypothetical. Token budget exhaustion is the #1 silent failure mode in production AI coding tools. Unlike compilation errors that fail loudly, running out of context mid-task produces subtle degradation: the agent "forgets" earlier constraints, generates code that conflicts with files it saw 80k tokens ago, or silently drops test coverage because the examples scrolled out of the window.

The Iron Triangle: Coverage vs. Fidelity vs. Latency

Every context window decision lives at the intersection of three forces:

**Coverage** is how much of the codebase you include. Sending 200 related files gives the agent a complete picture but consumes your budget fast. Narrow context (just the 5 files you think matter) runs cheap but the agent can't see the dependency that breaks your build.

**Fidelity** is how much detail you preserve per item. Full file contents are expensive but unambiguous. Summaries like "database client with connection pooling" save tokens but lose the actual connection string format your validation logic depends on.

**Latency** is the time cost of large prompts. A 100k token context takes 8-12 seconds just to process before the LLM starts generating. Tight iteration loops—plan, edit, compile, retry—demand faster feedback than "wait 10 seconds to see if the agent remembered your constraint."

You cannot optimize all three. Choosing where to compromise defines your system's behavior under pressure.

Tiered Context: Send What's Essential, Summarize the Rest

The practical middle ground is **tiered inclusion**. Not every file needs the same fidelity.

Start with the **anchor files**—the 3-5 modules the agent will directly edit. Send these in full, with line numbers and current state. This is your ~15-25k token baseline.

Next, identify **direct dependencies**: imports the anchor files use, interfaces they implement, test utilities they call. For these, send a **signature view**: public API surface, type definitions, docstrings. Skip private helper functions and implementation details. A 400-line class compresses to a 40-line signature consuming ~300 tokens instead of 3,000.

Finally, add **ambient context**: related files that provide conceptual grounding but won't be edited. Use structured summaries:

File: src/db/migrations/0042_add_audit_logs.sql

Purpose: Adds audit_logs table with user_id, action, timestamp

Constraints: user_id must reference users.id, action is ENUM type

Modified: 2024-11-03

This gives the agent enough to avoid breaking invariants (don't generate code that assumes audit_logs.action is a string) without burning 5k tokens on the full migration DDL.

Dynamic Pruning When You Hit the Ceiling

Even with tiering, you'll hit limits. A refactor touching 60 files exceeds any reasonable budget. The naive solution—truncate the oldest context—creates the "amnesia bug" where the agent's 50th edit contradicts guidance from the 5th.

**Compression on collision** is more robust. When token pressure forces a choice, replace entire blocks with *executable facts*:

• "All database clients in `src/services/*` follow the pattern: instantiate with `getConnection()`, wrap queries in `withTransaction()`, call `.close()` in finally blocks."
• "Test files under `tests/integration/` require environment variables `TEST_DB_URL` and `AWS_LOCALSTACK_ENDPOINT`."

These compressed rules are **checkable**. The agent can generate code, the validation loop can verify the pattern is present, and retry with the specific rule re-emphasized if violated. Contrast this with dropping the files entirely and hoping the LLM "just knows" your conventions.

In Goatfied's architecture, every loop iteration re-plans context budget: after the compile step, we measure what tokens the error messages and compiler output consumed, then adjust the next retry's context to preserve room for diagnostic data. If you sent 80k tokens in round one and got back 15k of error spans, round two's prompt drops to 65k of code context so the full error details fit.

When Fidelity Beats Coverage: The Shared Interface Example

Assume you're changing a `Logger` interface used in 120 files. **High-coverage / low-fidelity** would send one-line summaries of all 120. **Low-coverage / high-fidelity** sends the full implementation of the 5 most complex consumers.

In practice, the latter wins. Why? Because the failure mode for underspecified changes is *silent divergence*. If you tell the LLM "update Logger calls in 120 files" with only summaries, it will generate syntactically valid code that compiles but misses semantic nuances—like that `logger.error()` in the payment service must include a `transaction_id` field for audit compliance, while other services don't have that requirement.

Send fewer files with full context. The agent generates correct implementations for those 5, you extract the **pattern** from the diffs, then apply that pattern mechanically (via AST transforms or another agent pass) to the remaining 115. This is the "anchor and extrapolate" strategy: high confidence on a subset, then systematic rollout instead of 120 simultaneous low-confidence guesses.

Budgeting Validation Artifacts

The hidden token sink is validation feedback. A compile error across 12 files generates 8k tokens of output. Test failures with stack traces can hit 20k. If your initial context consumed 100k, you've now exceeded a 128k window before the agent sees the errors.

**Reserve 30% for feedback loops.** If your model's window is 128k, treat your code context budget as 90k. Plan prompts should stay under 60k so you have room for plan output, file proposals, and revision feedback without truncation.

In Goatfied, we enforce this with a **token budget contract** at the start of each agent loop. The planner receives a remaining-token count and must allocate: X for code context, Y for proposed changes, Z for validation results. If the planner exceeds X, the loop rejects the plan before calling the LLM, forcing re-scoping rather than silently dropping context mid-stream.

Measuring What Actually Matters

Raw token counts lie. A 50k-token prompt with redundant imports and boilerplate wastes budget. A 50k prompt with dense type definitions and error-prone edge cases is efficient.

Track **tokens per successful edit**. If you consistently burn 80k tokens to fix a 20-line function, your context strategy is mis-targeted. Compare across task types: renaming a function should cost <10k; adding a feature with new endpoints might justify 60k.

Second, measure **context-change correlation**. When a retry succeeds after initial failure, diff the token allocation between attempts. If you dropped 15k of unrelated test mocks and kept 5k of the actual interface definition, that's signal. Build heuristics: "For interface refactors, prioritize callers in `src/` over `tests/`, 3:1 ratio."

Related Reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [Benchmarks playbook #8: practical Goatfied tactics for shipping PRs](/blog/goatfied-benchmarks-playbook-8)