Latency Budgets For Ghost Text Experience Implementation Guide: practical systems guide

Ghost text—the grayed-out code suggestion that appears inline as you type—feels magical when it's instant and frustrating when it lags by even 200ms. The difference between a developer accepting a suggestion and ignoring it often comes down to whether the completion arrived before they finished typing the line themselves. This guide walks through the specific latency budgets, measurement strategies, and architectural choices that make ghost text feel responsive in production.

Why latency budgets matter more than total response time

A ghost text system that averages 150ms but spikes to 800ms every tenth request will feel worse than one that consistently delivers at 250ms. Users build muscle memory around a predictable cadence—they start typing, pause for a fraction of a second, and either accept or reject. Unpredictable latency breaks that rhythm and trains developers to stop trusting the suggestions.

The budget you need depends on what triggers the completion. Keystroke-triggered suggestions need to arrive within 100–150ms to feel instantaneous. Tab-triggered completions (where the user explicitly asks for help) can tolerate 300–400ms before frustration sets in. Contextual suggestions that appear after a pause (like when the cursor sits idle for 500ms) have more room—up to 600ms total—because the user isn't actively typing.

Measuring what users actually experience

Server-side metrics will lie to you. Your backend might report p95 latency of 120ms, but users experience the full round trip: network overhead, TLS handshake amortization, client-side debouncing, rendering time, and any queuing on either end.

Instrument the client. Measure from the moment you decide to request a completion to the moment the ghost text renders in the editor DOM. Break this into segments:

const timings = {

  debounceWait: debounceEnd - keystrokeTime,

  requestSent: fetchStart - debounceEnd,

  networkRTT: responseReceived - fetchStart,

  modelInference: responseReceived - requestSent - networkLatency,

  rendering: ghostTextVisible - responseReceived

};

Aggregate these client-side and bucket by user region, connection type, and model size. You'll often find that network variance swamps inference time for users on mobile connections or far from your edge, while enterprise customers on stable fiber see inference dominate.

At Goatfied, we measure completion latency as part of the broader agent loop where suggestions must pass compile and lint gates before surfacing. This adds 50–100ms over raw LLM output but catches a category of bad suggestions that would otherwise waste user attention. The tradeoff makes sense when the completion quality improves enough to increase acceptance rate—one avoided rejection is worth several hundred milliseconds of wait.

Allocating your budget across the pipeline

If your target is 150ms end-to-end for keystroke-triggered completions, a realistic breakdown might look like:

• **Debounce/batching**: 20–30ms. You need enough wait to avoid firing on every keystroke (expensive and chatty), but short enough that users don't perceive lag.
• **Network to backend**: 20–40ms for users near your edge, 60–120ms for distant or mobile users. Accept that you can't control this; optimize by moving compute closer or pre-establishing connections.
• **Model inference**: 40–80ms for small tuned models on dedicated GPU instances, 100–200ms for larger general-purpose models or shared infrastructure.
• **Post-processing and validation**: 10–30ms for syntax checks, formatting, and light filtering. If you run compile or lint gates like we do, reserve 50–100ms here.
• **Rendering**: 10–20ms to insert ghost text into the editor's virtual DOM and trigger a paint.

If your total exceeds the budget, cut from the middle of the pipeline first. Switch to a smaller or quantized model, reduce context window size, or skip validation for non-critical suggestions. Debounce time and rendering are harder to compress without degrading UX.

Caching and speculation strategies

Caching completions for repeated contexts can shave 80–100ms off hot paths, but naive caching backfires. Cache keys must include enough context—file type, surrounding lines, imported symbols—to avoid serving stale or wrong suggestions. A cache hit that gives the user something irrelevant is worse than a cache miss.

Speculative pre-fetching works well for predictable patterns. If a user types `function `, you can predict they'll soon need a completion and start inference before the debounce timer expires. The risk is wasted compute if they backspace or switch files. Limit speculation to high-confidence triggers and kill in-flight requests aggressively when the user's intent changes.

Goatfied's agent loop design naturally supports speculation because each suggestion is validated against compile/lint constraints before display. We can optimistically generate several candidates in parallel, then select and present the first that passes validation, discarding the rest. This parallelism often masks 30–50ms of inference time without increasing perceived latency.

Handling degraded conditions gracefully

Your system will occasionally miss the latency budget—overloaded GPU instance, temporary network congestion, cold start after inactivity. The worst response is to block the UI or show a spinner. Ghost text should appear or not appear; it should never interrupt the user's flow with loading states.

Set hard timeouts client-side. If a completion hasn't arrived within 200ms (or whatever your budget is plus a small margin), cancel the request and continue without a suggestion. Retry logic is tempting but usually wrong—by the time the retry completes, the user's context has changed and the completion is irrelevant.

Fallback to simpler models or local heuristics can bridge the gap. A deterministic snippet engine or tree-sitter-based completion might not be as smart as an LLM, but it delivers in 5–10ms and never times out. Serve the fast mediocre suggestion rather than gambling on the slow great one.

Instrumentation for continuous tuning

Latency budgets aren't set once and forgotten. As your user base grows and shifts geographically, as models improve or grow in size, and as you add features like multi-file context or lint gates, the budget needs revisiting.

Track acceptance rate by latency bucket. If users accept 40% of suggestions that arrive in under 100ms but only 15% of those that take 250ms, you've found your quality-speed frontier. Monitor regression alerts on p75 and p95 latency, not just mean—tail latencies determine whether power users trust your tool or turn completions off entirely.

Correlate latency with session retention. Users who experience consistently low latency tend to keep the feature enabled and use it more often, compounding the value. A 50ms improvement that lifts acceptance rate by 5 percentage points can translate to measurably higher DAU/MAU ratios over months.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)