Measuring ghost-text latency honestly

Learn how to accurately measure AI code completion latency across all four components that affect the delay between typing and seeing suggestions.

Every AI coding assistant brags about "near-instant" completions, but your cursor sits frozen for three seconds while you wait for a closing bracket suggestion. Measuring ghost-text latency—the delay between stopping your keystrokes and seeing an inline completion—turns out to be harder than most vendors admit, and the published numbers rarely survive contact with your actual workspace.

The problem isn't malice. It's that latency has at least four distinct components, only one of which shows up in vendor dashboards, and the interaction between them changes based on file size, repository structure, and whether you're working in a fresh checkout or a codebase that's been edited forty times since breakfast.

Why p50 doesn't tell the story

Most latency claims quote a median: "p50 of 180ms." That number usually measures model inference time—how long the LLM takes to generate tokens once the prompt is already assembled and sent. It ignores:

Context assembly: scanning open files, recent edits, LSP symbols, and git history to build the prompt
Network round-trip: especially brutal on hotel WiFi or VPN connections
Client-side rendering: painting the ghost text in your editor, which can stall if you have heavy syntax highlighting or a dozen extensions fighting for the same render loop

In a small TypeScript file with ten imports, context assembly might take 40ms. In a React component that imports thirty barrel files from a monorepo, it can blow past 800ms before the first token even gets requested. The model might return in 200ms, but you've already been waiting a full second.

A true p50 would need to measure wall-clock time from your last keystroke to pixels on screen. When we instrumented Goatfied's agent loop to log every stage, we found that for files over 1,000 lines, context assembly accounted for 60–70% of perceived latency, and the variance was wild—p50 might be 300ms but p95 could spike to 2.1 seconds if the LSP was mid-index.

What actually matters: the typing threshold

The useful metric isn't "how fast can the system respond" but "does it respond before I've already typed the next three characters?" If you type at 80 WPM (pretty typical for experienced engineers), that's roughly four characters per second, or 250ms per character. A 400ms latency means the suggestion appears after you've already moved on; a 150ms latency lands while your hands are still paused.

This creates a brutal cutoff around 200–250ms. Anything faster feels immediate. Anything slower starts to feel like the tool is chasing you rather than anticipating, and you learn to ignore the gray text entirely.

The variance matters more than the mean. A tool that's reliably 300ms is easier to work with than one that alternates between 100ms and 900ms, because you can't build muscle memory around chaos.

Instrumentation that doesn't lie

If you want honest measurements, you need three timestamps:

1. Keystroke receipt: when the editor's key handler fires (not when the debounce timer expires—some tools wait 150ms before even starting)

2. Request sent: when the completion request leaves the client, context fully assembled

3. Render complete: when the ghost text is painted and visible

Log all three, along with file size, number of imports, and whether the LSP is idle or busy. Over a few hundred completions, you'll see patterns:


{

  file: "src/components/Dashboard.tsx",

  lines: 847,

  imports: 23,

  lsp_busy: true,

  debounce_ms: 150,

  context_assembly_ms: 620,

  network_ms: 190,

  inference_ms: 210,

  render_ms: 35,

  total_ms: 1205

}

That 1.2-second completion looks catastrophic, but the breakdown tells you the LSP was re-indexing and context assembly paid the price. If nine out of ten completions in that file are under 400ms, the p95 is the actionable number—because that's the experience that makes you stop trusting the tool.

The compile-first advantage

Goatfied's architecture—plan, constrain, edit, validate, retry—front-loads more work than a pure ghost-text system. We run tsc --noEmit or cargo check before showing you a diff, which sounds slower. But it changes the latency calculus in two ways:

First, we don't fire on every keystroke. You explicitly ask for a change ("add error handling to this function"), so there's no debounce ambiguity. The clock starts when you hit Enter, and you expect to wait a beat.

Second, the validation gate catches a category of completions that would've landed fast but wrong. A ghost-text tool might suggest import { useEffect } from 'react' in 180ms, but if your tsconfig enforces import type annotations and the model missed it, you've traded a fast suggestion for a lint error thirty seconds later. Goatfied's loop catches that in the validate step—it might take 400ms total, but the diff you see is guaranteed to compile.

The tradeoff is latency for correctness. For exploratory typing where you just want the closing bracket, ghost-text at 150ms wins. For "add a database migration and update the ORM schema," spending 600ms to know the migration is valid and the schema matches is a bargain.

Self-hosted numbers vs. managed

We offer both a managed cloud service and a self-hosted deployment (runs in your VPC or on-prem). The latency profiles are completely different:

Managed: network round-trip is typically 40–80ms from US/Europe to our inference cluster, plus 200–300ms model time. Context assembly happens client-side, so large repos pay the cost locally. Total p50 is usually 350–450ms for medium-sized files.
Self-hosted: network round-trip drops to <5ms (localhost or same-VPC), but you're running the model on your own GPUs. If you've provisioned a 4×A10 setup, inference is comparable. If you're sharing a single L4 across a team, inference can spike to 800ms under load.

The self-hosted win isn't raw speed—it's consistency. You control the instance size, you see the queue depth, and you can scale horizontally when the team doubles. The managed service is faster to onboard but harder to predict at p95.

What we don't know yet

Honest measurement also means admitting gaps. We don't currently track:

How latency correlates with acceptance rate (do engineers accept more suggestions at 200ms than 400ms? Probably, but we haven't instrumented it)
Whether slower, higher-quality suggestions (validated + compiled) train users to be more patient
How latency tolerance varies by language (does a Rust engineer wait longer because compile cycles already take seconds?)

These are hard experiments to run without A/B testing cohorts for weeks, and we're prioritizing correctness and reversibility over hyper-optimization. But if you're evaluating tools, ask vendors for the instrumentation, not just the headline number.

What to ask in a demo

When you trial an AI coding assistant, open your actual workspace—the 400k-line monorepo with barrel imports and a slow CI pipeline—and measure:

Time from keystroke to visible suggestion (use a screen recorder and count frames if you have to)
How often suggestions fail linting or type-checking
Whether latency degrades after the twentieth edit in a session (some tools leak memory or don't garbage-collect old context)

If the vendor quotes "p50 of 180ms," ask which stage that measures and whether it includes context assembly. If they can't answer, the number is probably inference-only and optimistic.