Latency Budgets For Ghost Text Experience Design Tradeoffs: practical systems guide

Ghost text—those grey, half-visible suggestions that appear as you type—feels like magic when it's fast and garbage when it's slow. The difference between a 50ms and a 300ms response isn't just subjective annoyance; it fundamentally changes whether developers treat the feature as part of their flow or an obstacle to route around. This post walks through the actual systems constraints that determine ghost text latency and how to design around them without pretending you can bend physics.

The perception threshold nobody talks about

Human motor control operates on surprisingly tight timescales. When you type a character and ghost text appears 50-80ms later, your brain registers it as instantaneous—the suggestion feels like it's completing *your* thought. Push that to 150-200ms and developers start perceiving lag. The suggestion now feels like it's *interrupting* rather than assisting.

The brutal part: these thresholds don't care about your architecture. A brilliant ML model that takes 400ms to generate perfect suggestions will feel worse than a dumb n-gram predictor that returns in 60ms. Users will disable the slow one and keep the fast one, even if the fast one is wrong more often. Speed isn't a feature; it's a precondition for the feature to exist at all.

This creates an uncomfortable design constraint: you can't just optimize for correctness. You need to pick a latency budget first—say, "P95 under 120ms"—and then figure out what level of model sophistication you can afford within that budget.

Where the milliseconds actually go

Breaking down a typical ghost text request path:

• **Network RTT to inference server**: 10-40ms for same-region cloud, 80-150ms cross-continent, 200ms+ if you're routing through a VPN or corporate proxy
• **Request queuing and scheduling**: 5-50ms depending on current load and whether you have dedicated capacity
• **Model inference**: 30-200ms for small completion models, 300-800ms for larger context-aware models
• **Response parsing and rendering**: 5-15ms client-side

The network component is often underestimated. If your editor pings an inference endpoint in us-east-1 and your user is in Tokyo, you've already burned 160ms on speed-of-light delays before the model even sees the request. This is why Cursor and similar tools route aggressively to regional endpoints and why some teams are experimenting with on-device inference despite the model quality tradeoffs.

Queuing is the sneaky one. During peak hours or when the model is handling a batch of longer requests, a ghost text query might sit in a queue for 40ms even though inference only takes 80ms. Your P50 might look fine while your P95 is unusable.

Designing within the budget: three strategies

**Strategy one: tiered model routing**. Route ghost text requests to a fast, small model for immediate feedback (target: 60ms). Simultaneously fire a request to a smarter, slower model (200ms budget). If the fast model returns first, show its suggestion. If the slow model returns before the user has accepted or ignored the fast suggestion, upgrade the ghost text in place. This keeps perceived latency low while still surfacing better suggestions when time allows.

async function getGhostText(context: EditorContext) {

  const fastPromise = inferFast(context);  // 60ms target

  const smartPromise = inferSmart(context); // 200ms target

  

  const first = await Promise.race([fastPromise, smartPromise]);

  renderGhostText(first);

  

  const second = await Promise.race([

    smartPromise,

    delay(150) // don't wait forever

  ]);

  if (second && !userHasActed()) {

    upgradeGhostText(second);

  }

}

**Strategy two: context size throttling**. Ghost text doesn't always need your entire 100KB file buffer. For inline completions, the last 2KB of context often performs nearly as well as the full context but cuts inference time in half. Maintain a fast path that sends minimal context and a slow path that sends full context, similar to tiered routing but trading model size for context size.

**Strategy three: client-side caching and prefetch**. When a developer opens a file, prefetch likely completions for common patterns in that file type. Cache aggressively at the line level—if they're on line 47 and haven't modified line 46, reuse any ghost text you've already computed for this position. This doesn't eliminate latency but shifts it to moments when the user isn't waiting.

The compile-first constraint

At Goatfied, we add an extra wrinkle: ghost text suggestions go through the same compile/lint/test gates as explicit edits. This means we can't just return syntactically plausible completions; they need to actually compile in the current context. The naive approach—generate suggestion, then validate—adds 50-150ms to your latency budget. That often exceeds the perceptual threshold.

The workaround is to constrain generation rather than validate after the fact. Our agent loop runs a planning phase that understands the type context, available symbols, and current imports before generating any code. This means the suggestions that reach the user are much more likely to compile, but it requires rethinking the generation pipeline. You're trading some model flexibility for deterministic correctness and tighter latency bounds.

In practice: a 120ms ghost text response that compiles beats a 90ms response that produces red squiggles 40% of the time. Developers learn to distrust the fast-but-broken tool and stop accepting its suggestions, even on the 60% of cases where it's fine.

Regional deployment realities

If you're serving ghost text at scale, you eventually need inference capacity in at least three regions (US, EU, Asia-Pacific) to keep network latency under control. The hard question: do you replicate your full model stack everywhere or run a lightweight fast model globally and keep the heavy models in one or two regions?

The cost-latency tradeoff is sharp. Running full model capacity in five regions might cost 3-5x more than a two-region deployment, but it keeps your P95 latency 100ms lower. For a paid product, that's often worth it. For a freemium tool trying to control costs, you might accept higher P95 latency for free-tier users and prioritize the paid users with regional routing.

Self-hosted deployments sidestep some of this—if the inference runs on the user's machine or their private cloud, network RTT drops to near zero—but you're constrained by their hardware. A developer on a 2019 MacBook might see 200ms inference times where your cloud GPU completes in 80ms.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)