Latency budgets for ghost text completions

Learn how to set and maintain latency budgets for ghost text code completions to keep suggestions fast enough for developer acceptance.

Ghost text—the gray inline suggestion that appears as you type—feels magical when it's fast and infuriating when it lags. The difference between a 50ms and 300ms response isn't subjective annoyance; it fundamentally changes whether developers treat suggestions as part of their flow or obstacles to route around. This guide walks through the systems constraints that determine ghost text latency and how to design around them in production.

The perception threshold that determines acceptance

Human motor control operates on tight timescales. When you type a character and ghost text appears 50–80ms later, your brain registers it as instantaneous—the suggestion feels like it's completing your thought. Push that to 150–200ms and developers start perceiving lag. The suggestion now feels like it's interrupting rather than assisting.

The brutal part: these thresholds don't care about your architecture. A brilliant ML model that takes 400ms to generate perfect suggestions will feel worse than a simple n-gram predictor that returns in 60ms. Users will disable the slow one and keep the fast one, even if the fast one is wrong more often.

We've measured that engineers accept roughly 50% of sub-100ms suggestions but only 15% of 300ms+ completions, regardless of accuracy. The latency budget defines user experience more than model quality. Speed isn't a feature; it's a precondition for the feature to exist at all.

This creates an uncomfortable design constraint: you can't just optimize for correctness. You need to pick a latency budget first—say, "P95 under 120ms"—and then figure out what level of model sophistication you can afford within that constraint.

Breaking down the latency budget

A ghost text interaction has roughly five phases, each with distinct latency characteristics:

1. **Keystroke capture to request dispatch** (1–10ms): Editor event handling, debounce logic, context serialization. Usually negligible unless you're serializing massive files.

2. **Network round-trip** (10–200ms): A self-hosted model on localhost adds 5ms; a round-trip to a US-East datacenter from Europe burns 80–120ms before inference starts. Cross-continent requests can hit 200ms+ if routing through VPNs or corporate proxies.

3. **Model inference** (50–500ms): Smaller models (7B parameters) on good GPUs produce first-token in 50–80ms; larger models or high contention stretch this to 300ms+. Context size affects this linearly or worse.

4. **Streaming decode and display** (5–50ms): Receiving tokens over SSE/WebSocket, rendering incrementally. Well-optimized editors handle this in single-digit milliseconds per batch.

5. **Invalidation handling** (variable): If the user keeps typing while ghost text is in-flight, you need to cancel, re-request, or discard stale completions.

The practical budget for "feels instant" ghost text is roughly 100–150ms total from final keystroke to first visible suggestion. If your target is 150ms end-to-end, a realistic breakdown might look like:

**Debounce/batching**: 20–30ms
**Network to backend**: 20–40ms for nearby users, 60–120ms for distant/mobile
**Model inference**: 40–80ms for small tuned models, 100–200ms for larger ones
**Post-processing and validation**: 10–30ms for syntax checks, 50–100ms if running compile/lint gates
**Rendering**: 10–20ms

If your total exceeds budget, cut from inference or validation first. Debounce time and rendering are harder to compress without degrading UX.

Failure pattern: debounce miscalibration

Debouncing delays requests until typing pauses. Too short (50ms) and you fire excessive requests mid-word; too long (500ms) and the system feels unresponsive. Fixed debounce creates problems: rapid backspace-and-retype generates different patterns than typing prose in a docstring.

Fixed debounce either fires too eagerly—wasting inference on incomplete context—or too conservatively, missing opportunities where the user genuinely paused.

**Mitigation**: Adaptive debounce that shortens after certain keystrokes (`.` or `(` often signal completion points) and lengthens during rapid deletion. Track inter-keystroke intervals over a sliding window. Goatfied's agent loop uses context-aware triggers rather than pure time-based debounce, firing suggestions when syntactic boundaries indicate likely completion points (end of statement, closing brace).

Failure pattern: inference queue starvation

Cloud-hosted models often serve requests through shared inference queues. When load spikes, queuing delay dwarfs actual inference time. Your request travels in 40ms, sits in queue for 300ms, gets a 60ms response—total 400ms, well past acceptable threshold.

This manifests as inconsistent latency: sometimes ghost text appears instantly, sometimes it takes half a second for identical operations. Users notice variance more than average.

**Mitigation**: Prioritize ghost text requests with dedicated capacity or separate queue tiers. If self-hosting, reserve GPU memory/batch slots exclusively for low-latency interactive completions, even at slight throughput cost for batch workloads. If using managed services, configure request priorities or timeouts so ghost text fails fast rather than queueing indefinitely. A 200ms timeout with no result beats a 500ms stale suggestion.

Set hard timeouts client-side. If completion hasn't arrived within your budget plus small margin, cancel the request and continue without a suggestion. By the time a retry completes, the user's context has changed and the completion is irrelevant.

Failure pattern: context explosion

Longer prompts increase inference time linearly or worse. Sending entire file history plus open tabs can push prompt length past 8K–16K tokens, adding 100–200ms even on fast hardware.

The temptation is "more context = better suggestions," but ghost text has different constraints than long-form generation. A user typing `for i in range(` doesn't need full module history; they need immediate surrounding scope and maybe recent imports.

**Mitigation**: Context pruning that balances relevance and speed. Send only current function/class plus small sliding window of recent edits (last 50–100 lines). Use retrieval-augmented context (vector search) only for multi-line completions or explicit pauses (1+ second idle). For single-line ghost text, aggressive pruning to <2K tokens keeps inference predictable.

Score context relevance with:

**Recency**: Lines within 50 above/below cursor position score highest
**Import paths**: If current line references `import { foo }`, include `foo`'s definition
**Structural scope**: If cursor is inside a function, include signature and docstring but prune unrelated peers

Failure pattern: invalidation race conditions

User types `function getName(`, ghost text starts streaming `user)`, but the user has already typed `userId)` before the suggestion appears. Now the editor must discard the stale completion, show it anyway (confusing flicker), or merge intelligently (complex, bug-prone).

Poor invalidation creates "fighting the ghost" where suggestions appear after they're irrelevant, and users learn to ignore the feature entirely.

**Mitigation**: Tag every request with exact buffer state (file content hash + cursor position). When response arrives, validate that buffer state hasn't changed. If it has, discard immediately without rendering. For streaming completions, cancel in-flight requests on any keystroke and restart. This burns some inference budget but keeps UI coherent.


async function getGhostText(context: EditorContext) {

  const requestId = generateId();

  const bufferHash = hashBuffer(context.buffer);

  

  const response = await fetchCompletion(context, requestId);

  

  if (hashBuffer(context.buffer) !== bufferHash) {

    // Buffer changed, discard

    return null;

  }

  

  renderGhostText(response);

}

Tiered model routing and progressive rendering

A single model target creates brittleness. If your primary endpoint is slow or down, ghost text disappears entirely. Production systems use tiered fallbacks:

1. **Hot tier**: Small, fast model (1B params) with <30ms inference. Always available, lower quality.

2. **Warm tier**: Medium model (7B params) with 50–80ms inference, cached contexts. Primary choice.

3. **Cold tier**: Larger model or external API, 150–300ms. Used only when user explicitly requests or warm tier overloads.

Route ghost text to fast, small model for immediate feedback (60ms target). Simultaneously fire request to smarter, slower model (200ms budget). If fast model returns first, show its suggestion. If slow model returns before user has accepted or ignored the fast suggestion, upgrade ghost text in place. This keeps perceived latency low while surfacing better suggestions when time allows.


async function getGhostText(context: EditorContext) {

  const fastPromise = inferFast(context);  // 60ms target

  const smartPromise = inferSmart(context); // 200ms target

  

  const first = await Promise.race([fastPromise, smartPromise]);

  renderGhostText(first);

  

  const second = await Promise.race([

    smartPromise,

    delay(150) // don't wait forever

  ]);

  if (second && !userHasActed()) {

    upgradeGhostText(second);

  }

}

Progressive rendering shows hot tier suggestion immediately, then swaps in warm tier result if it arrives within 100ms. Users see something fast, and it upgrades seamlessly if the better model finishes in time.

Request pipelining and speculative context fetching

Most implementations wait until user pauses typing to fire completion request. By then, you've burned 50–100ms of idle time. Pipelining starts fetching context and warming caches while user is still typing.


let speculativeContext = null;



editor.onKeystroke(async (event) => {

  // Fire early, cancel if user keeps typing

  speculativeContext = fetchContext(

    editor.getBuffer(),

    event.cursorPosition

  );

});



editor.onPause(async (pauseEvent) => {

  const ctx = await speculativeContext;

  const completion = await inferWithContext(ctx, pauseEvent.cursorPosition);

  editor.showGhost(completion);

});

Speculative fetching cuts 20–40ms from critical path because parsing buffer, loading symbols, and priming KV cache happen in parallel with typing. You pay small cost in wasted compute when user changes direction mid-word, but p50 latency improvement justifies it.

Speculative pre-fetching works well for predictable patterns. If user types `function `, predict they'll soon need completion and start inference before debounce expires. Limit speculation to high-confidence triggers and kill in-flight requests aggressively when intent changes.

The compile-first constraint

At Goatfied, ghost text suggestions go through the same compile/lint/test gates as explicit edits. This means we can't just return syntactically plausible completions; they need to actually compile in current context. The naive approach—generate suggestion, then validate—adds 50–150ms to latency budget, often exceeding perceptual threshold.

The workaround is to constrain generation rather than validate after the fact. Our agent loop runs a planning phase that understands type context, available symbols, and current imports before generating code. This means suggestions that reach users are much more likely to compile, but it requires rethinking the generation pipeline—trading some model flexibility for deterministic correctness and tighter latency bounds.

In practice: a 120ms ghost text response that compiles beats a 90ms response that produces red squiggles 40% of the time. Developers learn to distrust fast-but-broken tools and stop accepting suggestions, even on the 60% of cases where they're fine.

We handle this by showing ghost text immediately without blocking on linting, running lint asynchronously in a worker thread, and if linted ghost text fails validation within 200ms, hiding it and showing next-best suggestion. This keeps UI responsive while maintaining compile-first rigor. The tradeoff is occasional flicker if suggestion fails lint, but engineers prefer fast feedback over waiting for perfect-but-slow validation.

Regional deployment and self-hosted latency control

If you're serving ghost text at scale, you need inference capacity in at least three regions (US, EU, Asia-Pacific) to keep network latency under control. The hard question: replicate full model stack everywhere or run lightweight fast model globally and keep heavy models in one or two regions?

The cost-latency tradeoff is sharp. Running full capacity in five regions costs 3–5x more than two-region deployment but keeps P95 latency 100ms lower. For paid products, often worth it. For freemium tools controlling costs, accept higher P95 for free-tier users and prioritize paid users with regional routing.

SaaS ghost text services introduce network round-trip latency (20–80ms depending on region) and variable cold-start delays. Self-hosted inference gives you control over entire latency profile. Running small model on local GPU (even M1 MacBook) eliminates network overhead and provides consistent <50ms inference.

Goatfied's self-hosted option lets teams deploy the agent loop on their infrastructure with ghost text inference on local or private cloud GPUs. This matters for regulated industries where sending code to external APIs is non-compliant, but also for latency: a warm local model beats a cold remote API every time.

For hybrid setups, route simple completions (single-line variable names, closing braces) to local models and complex multi-line suggestions to cloud endpoints. Local model handles 80% of requests with <30ms latency; remaining 20% tolerate higher latency because they're more intentional.

Instrumentation for continuous tuning

Latency budgets aren't set once and forgotten. As user base grows and shifts geographically, as models improve or grow in size, and as you add features like multi-file context or lint gates, budgets need revisiting.

Server-side metrics will lie to you. Your backend might report p95 latency of 120ms, but users experience full round trip: network overhead, TLS handshake amortization, client-side debouncing, rendering time, queuing on either end.

Instrument the client. Measure from moment you decide to request completion to moment ghost text renders in editor DOM. Break into segments:


const timings = {

  debounceWait: debounceEnd - keystrokeTime,

  requestSent: fetchStart - debounceEnd,

  networkRTT: responseReceived - fetchStart,

  modelInference: responseReceived - requestSent - networkLatency,

  rendering: ghostTextVisible - responseReceived

};

Aggregate client-side and bucket by user region, connection type, model size. You'll often find network variance swamps inference time for users on mobile connections or far from edge, while enterprise customers on stable fiber see inference dominate.

Track these metrics:

**p50, p90, p99 end-to-end latency**: From keystroke pause to ghost text visible
**Model inference time**: Isolated measurement of just model call
**Context fetch time**: How long buffer parsing and symbol lookup take
**Cache hit rate**: Percentage of requests served from warm KV caches

Set alerting thresholds: if p90 latency exceeds 150ms for more than five minutes, page on-call. Budget violations correlate directly with user frustration and feature abandonment.

Track acceptance rate by latency bucket. If users accept 40% of suggestions arriving in under 100ms but only 15% of those taking 250ms, you've found your quality-speed frontier. Correlate latency with session retention—users experiencing consistently low latency tend to keep features enabled and use them more often.

Caching and streaming strategies

Caching completions for repeated contexts can shave 80–100ms off hot paths, but naive caching backfires. Cache keys must include enough context—file type, surrounding lines, imported symbols—to avoid serving stale or wrong suggestions. A cache hit that gives irrelevant suggestions is worse than cache miss.

For ghost text, streaming usually wins over single-shot because perceived latency is time-to-first-token, not time-to-complete. A 300ms full completion feels slower than 80ms first-token + 220ms streaming. However, streaming adds coordination cost: handle cancellation, partial renders, case where streaming starts but user types over it mid-stream. If infrastructure can't reliably stream with <100ms first-token, single-shot may produce better experience by avoiding flicker.

Goatfied's agent loop design naturally supports speculation because each suggestion is validated against compile/lint constraints before display. We can optimistically generate several candidates in parallel, then select and present first that passes validation, discarding the rest. This parallelism often masks 30–50ms of inference time without increasing perceived latency.

[Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
[Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
[Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
[Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)