Latency Budgets For Ghost Text Experience Production Playbook: practical systems guide

Technical field guide on latency budgets for ghost text experience production playbook for teams building dependable AI coding workflows.

Ghost text—inline completions that appear as you type—feels magical when latency stays under 100ms. Beyond 200ms, engineers start ignoring suggestions or switching back to manual editing. This guide walks through the production systems that keep ghost text responsive at scale, covering everything from request pipelining to fallback tiering.

Why latency budgets matter more than model quality

A completion that arrives 500ms after you stop typing is rarely useful, even if the suggestion is perfect. Your mental model has already moved on. We've found that engineers accept roughly 50% of sub-100ms suggestions but only 15% of 300ms+ completions, regardless of accuracy. The latency budget defines the user experience more than the underlying model's capabilities.

Latency budgets force hard architectural decisions early. If you budget 80ms end-to-end for ghost text, you might have 10ms for request parsing, 15ms for context retrieval, 40ms for inference, and 15ms for response rendering. That 40ms inference window rules out larger models or forces you into speculative decoding, KV cache optimization, or prefix sharing. The budget shapes the system.

Request pipelining and speculative context fetching

Most ghost text implementations wait until the user pauses typing to fire a completion request. By then, you've already burned 50–100ms of idle time. Pipelining starts fetching context and warming caches *while* the user is still typing.

Here's the basic pattern:


let speculativeContext = null;



editor.onKeystroke(async (event) => {

  // Fire early, cancel if user keeps typing

  speculativeContext = fetchContext(

    editor.getBuffer(),

    event.cursorPosition

  );

});



editor.onPause(async (pauseEvent) => {

  const ctx = await speculativeContext;

  const completion = await inferWithContext(ctx, pauseEvent.cursorPosition);

  editor.showGhost(completion);

});

Speculative fetching cuts 20–40ms from the critical path because parsing the buffer, loading relevant symbols, and priming the KV cache happen in parallel with typing. You pay a small cost in wasted compute when the user changes direction mid-word, but the p50 latency improvement justifies it.

Tiered fallbacks and progressive rendering

A single model target creates brittleness. If your primary endpoint is slow or down, ghost text disappears entirely. Production systems use tiered fallbacks:

1. **Hot tier**: Small, fast model (e.g., 1B params) with <30ms inference time. Always available, lower quality.

2. **Warm tier**: Medium model (7B params) with 50–80ms inference, cached contexts. Primary choice.

3. **Cold tier**: Larger model or external API, 150–300ms. Used only when user explicitly requests completion or warm tier is overloaded.

Progressive rendering shows the hot tier suggestion immediately, then swaps in the warm tier result if it arrives within 100ms. The user sees *something* fast, and it upgrades seamlessly if the better model finishes in time. This prevents the "blinking ghost text" problem where suggestions flicker in and out.

Context window pruning and relevance scoring

Sending the entire file buffer to the model on every keystroke is wasteful. Most tokens are irrelevant to the current completion. Effective pruning cuts inference cost and latency.

We score context relevance with:

**Recency**: Lines within 50 above/below cursor position score highest.
**Import paths**: If current line references `import { foo }`, include `foo`'s definition.
**Structural scope**: If cursor is inside a function, include function signature and docstring but prune unrelated peer functions.

A simple pruning heuristic:


def prune_context(buffer, cursor_line, max_tokens=2048):

    lines = buffer.split('\n')

    scored = []

    for i, line in enumerate(lines):

        distance = abs(i - cursor_line)

        score = 1 / (1 + distance * 0.1)  # Exponential decay

        scored.append((score, line))

    

    scored.sort(reverse=True)

    pruned = ''.join([line for _, line in scored[:max_tokens // 20]])

    return pruned

This cuts a 4000-token file down to 2000 tokens, halving inference time without sacrificing local context quality. More sophisticated systems add semantic embedding lookups to pull in distant but relevant definitions.

Measuring and enforcing budgets in production

Declaring a latency budget is easy; enforcing it requires instrumentation. Track these metrics:

**p50, p90, p99 end-to-end latency**: From keystroke pause to ghost text visible.
**Model inference time**: Isolated measurement of just the model call.
**Context fetch time**: How long buffer parsing and symbol lookup take.
**Cache hit rate**: Percentage of requests served from warm KV caches.

Emit these as structured logs:


{

  "event": "ghost_completion",

  "latency_ms": 87,

  "breakdown": {

    "context_fetch": 12,

    "inference": 52,

    "rendering": 8

  },

  "cache_hit": true,

  "model_tier": "warm"

}

Set alerting thresholds: if p90 latency exceeds 150ms for more than five minutes, page on-call. Budget violations correlate directly with user frustration and feature abandonment.

Compile gates and ghost text interaction

Ghost text and compile-first workflows intersect awkwardly. If the editor is running continuous background linting (as Goatfied does in the agent loop: plan → constrain → edit → validate → retry), a ghost suggestion that introduces a type error will trigger immediate feedback. This is good—it prevents broken suggestions from propagating—but it adds latency if the linter runs synchronously.

We handle this by:

1. Showing ghost text immediately without blocking on linting.

2. Running lint asynchronously in a worker thread.

3. If the linted ghost text fails validation within 200ms, hide it and show the next-best suggestion.

This keeps the UI responsive while maintaining compile-first rigor. The tradeoff is occasional flicker if a suggestion fails lint, but engineers prefer fast feedback over waiting for perfect-but-slow validation.

Self-hosted latency control

SaaS ghost text services introduce network round-trip latency (20–80ms depending on region) and variable cold-start delays. Self-hosted inference gives you control over the entire latency profile. Running a small model on a local GPU (even an M1 MacBook) eliminates network overhead and provides consistent <50ms inference.

Goatfied's self-hosted option lets teams deploy the agent loop on their own infrastructure, with ghost text inference running on local or private cloud GPUs. This matters for regulated industries where sending code to external APIs is non-compliant, but it also matters for latency: a warm local model beats a cold remote API every time.

For hybrid setups, route simple completions (single-line variable names, closing braces) to local models and complex multi-line suggestions to cloud endpoints. The local model handles 80% of requests with <30ms latency; the remaining 20% tolerate higher latency because they're more intentional.

[Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
[The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)