Context Window Budgeting Under Token Pressure Production Playbook: practical systems guide

# Context Window Budgeting Under Token Pressure: Production Playbook

When your AI coding agent hits the context limit mid-PR, the edit stops cold. The diff you were about to merge becomes partial, the test suite never runs, and the agent loop breaks. Token pressure isn't an abstract scaling concern—it's a reliability failure that surfaces in production automation workflows, usually at the worst moment.

This playbook covers how to budget context windows when you're building systems that generate code under real constraints. We'll focus on the engineering decisions that keep agent loops stable when token counts climb: what to measure, where to gate, and how to structure edits so you stay within budget without sacrificing correctness.

Why context limits break automated workflows

Context windows determine how much source code, test output, lint feedback, and conversation history an LLM can hold in a single request. Modern models offer 128k, 200k, or more tokens, but that ceiling is fixed. Once you exceed it, the model either truncates silently (losing critical context) or fails outright (halting your automation).

In a typical Goatfied agent loop—plan, constrain, edit, validate, retry—the context accumulates at each step. The original file content, the plan description, the proposed diff, the compiler error, the lint output, the test failure trace—all of these must fit together. If your agent is editing a 3,000-line module and the test suite dumps 10,000 lines of output, you can blow past 200k tokens before the second retry.

The failure mode is subtle. The model stops mid-edit, produces a syntactically incomplete diff, or omits the test invocation entirely. Your CI gate catches the broken code, but the agent can't recover because it no longer has room to re-read the error and adjust. The PR stalls.

Measuring token consumption in the agent loop

Start by instrumenting every step of your automation to log token counts. For each LLM call, record the input token count (prompt + context) and output token count (generated code/plan). Track this per-step: planning, editing, validating.

A baseline measurement might look like this:

• **Plan step:** 5k tokens (file summary + task description)
• **Edit step:** 40k tokens (full file + plan + diff instructions)
• **Validation step:** 60k tokens (file + diff + compiler output + lint results)
• **Retry step:** 80k tokens (everything above + error analysis)

If your model has a 128k context window, you're already at 62% capacity after one retry. Two retries could exceed the limit.

Log these per-PR or per-task. Export to a time-series database or append to structured logs so you can query percentiles. You want to know: what's the p95 token consumption for edits that touch files over 2,000 lines? How often do retries push you over budget?

Constraining scope before the edit

The most effective way to stay under budget is to shrink the problem before the agent starts editing. This means breaking large files into smaller edits, or narrowing the agent's focus to a specific function or class.

Goatfied's "plan → constrain" step is designed for this. Before generating a diff, the agent identifies which functions or modules need to change. You can enforce a maximum edit scope—say, 500 lines of code per diff—at the constraint stage. If the plan requires changes across 1,500 lines, split it into three sequential diffs.

Each diff gets its own compile/lint/test gate. This keeps the context window small for each individual edit. The tradeoff: you introduce sequencing dependencies. If diff two assumes diff one is already applied, the agent must apply them in order. But the benefit is clear—each step stays well within budget, and you can retry individual diffs without re-reading the entire file history.

Streaming and incremental validation

For edits that must touch large files, stream the validation feedback incrementally rather than loading everything at once. Run the compiler first. If it fails, feed only the compiler error back to the agent. Don't append the full lint report and test output until the code compiles.

This keeps the validation step's token count proportional to the severity of the error. A single type mismatch might cost 2k tokens to describe; a clean compile followed by one failing test might add 8k. But you avoid dumping 50k tokens of cascading errors from a file that doesn't even parse.

Goatfied's validate-then-retry loop implements this by gating each feedback type. Compile errors block lint runs. Lint failures block test runs. Each gate is a checkpoint where the agent can consume feedback and retry without accumulating the full error stack.

Truncating history when retries accumulate

After three or four retries, the conversation history alone can consume tens of thousands of tokens. Each retry includes the previous error, the attempted fix, and the new error. This compounds quickly.

Set a maximum retry depth—typically three retries per edit. If the agent hasn't converged by then, fail the task and surface it for human review. Before you hit that limit, consider truncating older conversation turns from the context. Keep the most recent error and the current file state, but drop the first retry's full trace.

This is a judgment call. Sometimes the first error contains critical information the agent needs later. But in practice, if the agent hasn't solved the problem in two retries, re-reading the first error rarely helps. Truncation buys you room for the third attempt.

Code examples: budget gates in automation

Here's a simplified token gate you might implement before each LLM call:

MAX_CONTEXT_TOKENS = 120_000  # Leave headroom under model limit



def check_budget(prompt_tokens, history_tokens, buffer_tokens=8000):

    total = prompt_tokens + history_tokens + buffer_tokens

    if total > MAX_CONTEXT_TOKENS:

        raise ContextBudgetExceeded(total, MAX_CONTEXT_TOKENS)

Before invoking the model, count tokens for the file content, the diff, the error feedback, and the conversation history. If the sum exceeds your threshold, either truncate history or split the edit. Don't let the model call proceed if you're already over budget—it will fail or produce incomplete output.

Budgeting for self-hosted vs. managed deployments

Token budgets differ between self-hosted models and managed API endpoints. Managed providers (OpenAI, Anthropic) enforce strict context limits at the API level. Self-hosted models may allow you to configure larger contexts, but you pay in latency and GPU memory.

If you're running Goatfied self-hosted with a local model, profile the memory and inference time at different context sizes. A 200k token context might fit in VRAM but take 10 seconds per inference, breaking interactive feedback loops. A 128k limit with 2-second inference might be more practical for automation that retries frequently.

When to fail fast instead of retry

Not every token overrun is worth retrying. If the agent blows the budget on the initial edit (before validation), the problem is likely scope, not execution. Fail the task immediately and split it into smaller edits. Retrying with the same scope will just hit the limit again.

Conversely, if the budget overrun happens on the third retry after accumulating long error traces, that's a signal the agent is stuck. Fail and escalate to human review. Don't burn more tokens on a converging loop.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [Benchmarks playbook #8: practical Goatfied tactics for shipping PRs](/blog/goatfied-benchmarks-playbook-8)