Right-sizing GPUs for code completion workloads

Learn how to match GPU specs to your code completion request patterns, latency requirements, and team size without overspending on idle compute capacity.

Code completion models run inference hundreds or thousands of times per day per developer. Get the GPU wrong and you'll either blow your infrastructure budget on idle capacity or watch your team wait three seconds for every autocomplete suggestion. Neither is acceptable when you're trying to ship.

The challenge isn't finding the biggest GPU—it's matching compute to your actual request pattern. A 10-developer team hitting a shared completion endpoint behaves nothing like a 200-developer org where half the team works European hours. Your batching opportunities, latency requirements, and cost per token all shift dramatically.

Why code completion has different GPU requirements than chat

Chat interfaces tolerate latency. A developer asking "how do I parse this JSON?" will wait two seconds for a thoughtful answer. Code completion dies at 300 milliseconds. Beyond that threshold, the suggestion arrives after the developer has already typed the next token, making the whole interaction feel broken.

This latency constraint shapes everything:

Batch size stays small. You can't hold requests to fill a batch of 32 when the first request in the batch has already violated your latency budget. Most completion deployments run batch sizes of 1-4.
Model size matters less than you think. A 7B parameter model that fits entirely in a single GPU's VRAM will outperform a 13B model split across two GPUs for this workload, even if the 13B model is "better" by benchmark metrics.
Prefill dominates. Code completion means feeding the model everything from the current file (and often adjacent files) to predict the next 20-50 tokens. The time to process that context—prefill—is your primary latency component.

Compare this to a batch inference job processing thousands of GitHub issues overnight. There, you maximize throughput by packing huge batches and don't care if individual requests take five seconds. Completely different optimization target.

Request patterns shape utilization

A GPU running code completion sits idle most of the time. Developers think, switch files, attend meetings, take lunch. Even a highly active developer might send 500 completion requests across an 8-hour day—that's roughly one request per minute, with huge variance.

Three patterns to size for:

Solo or small team (1-15 developers). Traffic is bursty. You might see zero requests for 20 minutes, then eight concurrent requests when everyone pushes toward a deadline. A single GPU handles this fine, but you need enough VRAM for the model you want and enough compute to keep p95 latency under 200ms when requests do stack up. An NVIDIA L4 or T4 works well here—mid-range cost, sufficient memory for 7B models, and reasonable performance for sequential requests with occasional small batches.

Medium team (15-100 developers). Now you have enough concurrent users that batching becomes viable. If four requests arrive within 50ms of each other, you can process them together and amortize the prefill cost. This is where you graduate to something like an A10G or L40—better compute throughput for batch sizes of 2-8, and enough VRAM to keep multiple requests' KV caches resident.

Large deployment (100+ developers). Batching opportunities are consistent, you need redundancy anyway, and you're probably running multiple model variants (fast completion, slower but better inline suggestions, different models for different languages). Here you're comparing total cost of multiple smaller GPUs versus fewer large ones. Two A10Gs might cost less per month than one A100 while providing better fault tolerance.

VRAM determines what you can load, compute determines latency

A common mistake: picking a GPU based purely on memory size because "I need to fit a 13B model." Then wondering why inference is slow despite 24GB of free VRAM.

For a 7B parameter model in fp16:

Model weights: ~14GB
KV cache for 4K context: ~2GB per request
Activation memory: ~1-2GB depending on batch size

An L4 with 24GB VRAM can hold the model plus KV cache for 3-4 concurrent requests comfortably. But if you're running continuous batching and your p95 latency climbs above 400ms, the issue isn't memory—it's that the L4's compute throughput (about 120 TFLOPS) can't keep pace with your request rate.

Switching to an A10G (250 TFLOPS) with the same VRAM capacity would cut your latency in half for the same traffic pattern. Or you could stick with the L4 but drop to a 3B model, which requires less compute per token while still fitting in VRAM.

The tradeoff: smaller models complete faster but make less intelligent suggestions. You're balancing developer experience against infrastructure cost.

Measuring what matters

Before you commit to GPU SKUs, instrument three things:

p50, p95, and p99 latency, not averages. If your median request completes in 150ms but your p95 is 800ms, one in twenty completions will frustrate your developers. Track these separately for prefill and decode, since they stress different parts of the GPU.

Actual batch sizes achieved. Your batching logic might allow batches up to 16, but if you're only averaging 2.3 requests per batch in production, you're not getting the throughput you designed for. This tells you whether you need better traffic or different infrastructure.

GPU utilization over time, not point samples. A GPU showing 60% utilization averaged over a day might hit 100% for 90 seconds during standup when everyone opens their IDE, then idle at 5% for an hour. Those spikes are where latency breaks. Use a monitoring window of 30-60 seconds to catch them.

Example pattern in your Grafana dashboard:


histogram_quantile(0.95, 

  rate(completion_latency_seconds_bucket[5m])

)

If this metric trends above 0.3 (300ms) for more than a few minutes during work hours, you're undersized.

Compile gates catch GPU config drift

Goatfied's constraint system extends to infrastructure as code. When you define your GPU workload requirements, you can enforce them at compile time:


const completionGPU = new GPU({

  type: "nvidia-l4",

  memory: "24GB",

  constraints: {

    maxLatencyP95: 300, // milliseconds

    minThroughput: 50,  // requests/sec

  }

});

This won't prevent your traffic from spiking, but it prevents you from deploying a config that can't possibly meet your latency SLA. If someone tries to switch to a T4 (half the compute) without updating the latency constraint, the deployment fails before it reaches production.

The constraint isn't enforced at runtime—it's a compile-time check that your GPU spec can theoretically handle the load you've defined. You still need runtime monitoring, but you eliminate the entire class of problems where someone makes a "cost optimization" that unknowingly degrades developer experience.

When to scale horizontally versus vertically

You've hit your latency budget on your current GPU. Two paths:

Vertical: upgrade to a faster GPU. Makes sense when your p95 batch size is still 1-2 requests. A single A10G will beat two L4s for sequential traffic because there's no network hop and no request routing overhead. Simpler to operate, fewer failure modes.

Horizontal: add more of the same GPU. Better when you have consistent batching (p50 batch size ≥ 4) and want fault tolerance. Two L4s behind a load balancer cost about the same as one A10G but give you redundancy and make it easier to do rolling updates without downtime.

Most teams start vertical (one good GPU) then go horizontal (multiple replicas) as headcount grows. The crossover point is usually 30-50 active developers.

Self-hosted control, managed convenience

Goatfied's agent loop runs on either self-hosted infrastructure or managed cloud. For GPU workloads, self-hosted makes sense when you have existing GPU capacity, strict data residency requirements, or want precise control over batching and model serving logic.

The plan → constrain → validate cycle works identically in both modes. Your constraint definitions catch configuration errors before deployment, whether you're targeting your own Kubernetes cluster or Goatfied's managed GPU pool.

For air-gapped environments where you can't call external model APIs, self-hosted is the only option. You control the model weights, the inference runtime, and the network boundaries—critical for regulated industries or classified work.