GPU economics for self-hosted code models

Running your own code-generation models means trading subscription fees for infrastructure complexity. The breakeven point isn't obvious: a $20/month SaaS seat looks cheap until you're buying it for 50 engineers, but a self-hosted A100 rig carries fixed costs whether you use it or leave it idle. The real challenge isn't buying a GPU—it's sizing the cluster to match actual utilization patterns, picking models that fit your memory budget, and building orchestration that doesn't leave expensive hardware idle or oversubscribed.

Why code model inference breaks traditional GPU economics

Standard inference guidance assumes steady-state workloads. A customer-facing chatbot sees fairly predictable traffic curves. A recommendation engine processes events in a consistent stream. Code models serve spiky, latency-sensitive requests where a 4-second wait feels broken but a 400ms response feels instant.

Code completion traffic is bursty. Mornings after standup, right before lunch, late afternoon before commits. You'll see 2–3 concurrent requests for hours, then suddenly 15 simultaneous autocomplete calls when everyone's heads-down. A single engineer might send 5–10 requests per hour during active coding sessions. If you provision one GPU per person, utilization hovers around 2–5%.

You can't horizontal-autoscale GPUs like you would web servers. Spinning up a new GPU instance takes 2–5 minutes even with optimized AMIs and preloaded model weights. By the time your autoscaler reacts to a spike, your developers have already context-switched away or grabbed coffee. The only practical solution is overprovisioning, which means paying for capacity you're not using.

The cost structure compounds this. Cloud GPU pricing is ruthlessly linear: a half-idle H100 costs the same as a fully saturated one. Unlike CPU instances where you can rightsize to t3.medium and call it done, GPU options jump in large increments. You're choosing between a $2/hour T4 that can barely keep up with a single developer or a $4/hour L4 that handles three comfortably but wastes two-thirds of its capacity.

Measuring what actually matters: requests per developer-hour

Forget theoretical tokens/second. The metric that predicts your economics is requests per developer-hour in active coding sessions. Instrument your IDE or track API calls for a week. Most teams discover they average 15–40 completion requests per developer per active hour, with peaks hitting 80–100 during intense refactoring or when someone is learning a new codebase.

Each request's latency budget is ~500ms for inline completions and ~2s for chat-style interactions. Longer than that and developers lose flow state. This forces you into a quality-of-service calculation: can you serve your peak concurrent user count under latency SLA with X GPU capacity?

Start by measuring your actual workload. Pull a week's worth of API logs from your current code assistant (Copilot, Claude, or whatever you're using). Calculate tokens/day, peak request rates, and latency percentiles. If your team of 50 engineers averages 2M tokens/day with p95 latency under 800ms, you need capacity to handle ~23 tokens/second sustained, with burst headroom for morning standup spikes.

The ultimate unit economics metric is cost per million tokens generated, adjusted for quality. A model that generates code requiring 40% retry cycles isn't cheaper just because the instance costs less per hour—you've multiplied token volume by 1.4× and added latency. Instrument your stack to log wall-clock time per request, input/output token counts, validation outcomes (does the code compile? pass linting?), and retry/edit cycles before a valid diff.

GPU tiers and their practical limits

Map your workload to GPU memory and throughput. A 7B parameter model in float16 needs ~14GB VRAM just to load weights; 13B needs ~26GB; 34B needs ~68GB. Add KV cache overhead (roughly 1–2GB per 4k context window) and batch padding, and you're looking at:

**Consumer cards (RTX 4090, 4080):** 24GB VRAM on a 4090 runs 7B–13B models comfortably. Good for prototyping or small teams (5–10 devs) with bursty usage. Expect ~$1,600 per card, but consumer warranties don't cover 24/7 datacenter use. Power draw is 450W; you'll need proper PSUs and cooling.

**Datacenter cards (A100, H100, L40S):** An 80GB A100 handles 30B–70B models or serves multiple smaller models concurrently. List price is $10k–15k, but availability fluctuates. H100s double throughput but cost $25k+. L40S slots in at $7k with 48GB—often the sweet spot for code models where context windows matter more than raw FLOPs. An L40S comfortably handles 13B models in float16 or quantized 34B, serving 20–40 concurrent requests with dynamic batching.

**Cloud instances:** AWS `p4d.24xlarge` (8×A100 80GB) runs $32/hour on-demand, ~$10/hour with reserved capacity. Azure ND96amsr A100 v4 is similar. This works if your usage is genuinely bursty—spin up for CI/CD runs, tear down overnight—but reserved instances demand multi-month commits. A single A100 (40GB) costs roughly $3–4/hour on major clouds; GCP A100 40GB reserved 1-year runs ~$7k/month.

The key insight: **context length bottlenecks matter more than parameter count** for code tasks. A 13B model with 32k context on an L40S often outperforms a 70B model with 8k context on an A100 because code generation needs to see entire files, not just snippets.

Model size tradeoffs and quantization

**7B models** (CodeLlama-7B, DeepSeek-Coder-7B) fit comfortably in 16GB VRAM with room for KV cache. They run on consumer RTX 4090s or datacenter L4s. Quality is good enough for autocomplete and simple refactors but struggles with multi-file context or nuanced logic. A quantized 7B might deliver 60 tokens/sec on a single L4, enough to feel instant for most completions.

**13B–15B models** are the sweet spot for self-hosted production. They need 24–32GB VRAM depending on quantization (FP16 vs. int8). An RTX 6000 Ada (48GB) or A10G (24GB) handles one instance; an A100 (80GB) can run two with separate processes. Noticeably better at following instructions and handling 4k+ token contexts.

**34B+ models** require 48GB+ even with int8 quantization. You're looking at A100 80GB or H100 territory. The quality jump is real—better at architecture-level reasoning, understanding framework idioms—but inference is 2–3× slower per token. A 70B model on the same hardware drops to 15 tokens/sec—perceptibly slower but potentially more accurate. Reserve these for asynchronous tasks (PR review, test generation) where 1–2 second latency is acceptable.

Quantization to INT8 or INT4 can halve or quarter your memory footprint. A 13B model that needs 26GB in float16 fits in 13GB with int8, or 7GB with int4. In practice, int8 quantization (via `bitsandbytes` or `llm.int8()`) preserves >98% of original quality for most code tasks—autocompletion, docstring generation, simple refactors. int4 (GPTQ, AWQ) is rougher: you'll see more hallucinated imports, off-by-one logic errors, and degraded performance on complex multi-file edits.

Test this empirically. Take a representative eval set—100 code snippets your team actually wrote last month—and run them through both the full-precision and quantized model. Measure pass@1 compile rate, lint error rate, and subjective "would I accept this suggestion" scores. If int8 quantization drops your compile-first-try rate from 87% to 85%, that's probably acceptable. If int4 drops it to 72%, the memory savings aren't worth the frustration.

Batching and utilization: the real cost driver

A single engineer generating code interactively keeps a GPU at 5–15% utilization. The fix is request batching—queue up multiple edits, run inference on all of them, return results. Modern inference servers (vLLM, TensorRT-LLM, text-generation-inference) support **continuous batching**: new requests join in-flight batches dynamically instead of waiting for the entire batch to finish. This is non-negotiable. Static batching destroys tail latency.

When three engineers hit "generate" within 100ms of each other, the server runs one forward pass with batch_size=3 instead of three sequential passes. Throughput scales near-linearly up to batch sizes of 16–32, after which memory bandwidth becomes the bottleneck. This can push utilization to 40–60% during active development hours.

The KV cache stores attention keys/values from previous tokens so the model doesn't recompute context on every new token. A 4k context with 13B FP16 model eats ~2GB VRAM per request. With 48GB total, you can fit ~8 concurrent 4k-context requests plus the model weights. Monitor cache evictions—if you're thrashing, either reduce max context length or add more VRAM. PagedAttention (from vLLM) reduces memory fragmentation by splitting the KV cache into fixed-size blocks, increasing effective batch size by 20–40% without changing hardware.

Goatfied's agent loop naturally batches: the *plan* step generates multiple candidate changes, the *constrain* step filters them through type/lint rules, then *edit* applies them in parallel before *validate* runs tests. The validation gates catch bad output from quantized or smaller models before they propagate, but you still pay the latency cost of retry cycles.

But utilization still craters at night. Options:

• **Multi-tenant workloads:** Run code models during business hours, ML training jobs overnight. Requires tooling to schedule and kill jobs cleanly.
• **Spot/preemptible instances:** AWS Spot or GCP Preemptible can cut costs 60–80%, but you need graceful degradation when instances vanish mid-request. Works if your agent loop can checkpoint and retry. AWS g5.12xlarge (4× A10G) spot averages $3–5/hour—good for dev/staging; too risky for production without robust failover.
• **Regional load distribution:** If you're global, shift workloads to follow the sun. APAC engineers hit GPUs during their day, then EMEA takes over, then Americas. This assumes you can tolerate cross-region latency (50–150ms), which code generation usually can—compile/test gates take seconds anyway.

Cost comparison: realistic scenarios

**Scenario A: 30-engineer team, moderate usage (10 edits/dev/day)**

• SaaS ($20/seat/month): $7,200/year
• Self-hosted (1× L40S, $7k upfront + $200/month hosting): $9,400 first year, $2,400/year after
• Breakeven: 18 months, assuming 50% GPU utilization

**Scenario B: 150-engineer team, heavy usage (50 edits/dev/day)**

• SaaS ($30/seat/month for enterprise tier): $54,000/year
• Self-hosted (4× A100 80GB, $50k upfront + $1,200/month hosting + $60k/year ops headcount): $124,400 first year, $74,400/year after
• Breakeven: never (unless you value data control above cost savings)

**Scenario C: 150 engineers, compliance-driven, fine-tuned models**

• SaaS: Not viable (data can't leave infrastructure)
• Self-hosted (4× A100 + ops + fine-tuning pipeline): $124k/year baseline, but you're also avoiding potential regulatory penalties and getting model customization

The ops headcount is real—someone needs to patch kernels, rotate logs, debug CUDA OOMs, manage model updates. Budget 0.5–1 FTE for every 4 GPUs. For a team under 25 developers, these ops costs often exceed the GPU bill itself. You're paying a senior infra engineer to babysit model servers instead of building product. The break-even point is typically 50–100+ developers where the economies of scale justify dedicated ML platform work.

An alternative cost model for self-hosted on-prem: Buy RTX 6000 Ada ($6,800) or A10G ($2,500 used). One-time CapEx, 3-year depreciation. For a 20-person team, ~$340/dev over 3 years. If your utilization is spiky—most requests between 9am–5pm, near-zero overnight—you're paying for idle capacity. A single L40S at $1.50/hour × 24 hours = $36/day, even if it's only busy 6 hours. Contrast that with API pricing: if your 2M tokens/day costs $10–15 on a hosted service, self-hosting only wins if you can amortize fixed costs across higher volume or longer timelines.

Operational gotchas and failure modes

**VRAM leaks:** Long-running inference servers slowly leak memory. Restart them nightly or use container orchestration with health checks.

**Model version sprawl:** Engineers pin to "that one model that worked well last sprint." Suddenly you're serving six different checkpoints. Enforce deprecation timelines. Treat model weights like container images—test new versions in a canary environment (5% traffic), validate latency and quality on real traffic, then roll out.

**Cold start latency:** Lazy-loading models from disk adds 10–30 seconds to first request. Keep hot models in memory, but that reduces concurrency. Model startup takes 30–60 seconds (loading weights, warming up KV cache), so Kubernetes HPA based on request queue length is too slow. Better: keep 2–3 instances warm during business hours and scale up at known traffic peaks (9am, 1pm).

**Cold cache death spiral:** If your model's KV cache evicts between requests, every completion pays the full prefill cost. For a 34B model, prefill can be 3–5x slower than decode. Solution: keep a warm pool of cached contexts for recently active files. In a typical coding session, 30–40% of tokens in successive completions are identical prefix material (common imports, function signatures). Prefix caching turns that redundancy into a 2–3× effective speedup.

**Networking bottlenecks:** If your model server and dev environments are in different datacenters, RTT kills responsiveness. Co-locate or use edge PoPs. Self-hosting shifts the cost profile from per-token API fees to infrastructure + data transfer. If your code models run in us-east-1 but your CI/CD pipeline and artifact storage live in eu-west-1, cross-region egress can quietly consume 10–15% of your bill. A single 500MB model checkpoint download costs $0.045 in inter-region transfer; multiply by dozens of node scaling events per day. Colocate inference close to your source repositories and CI runners.

**Over-tuned models:** Fine-tuning a base model on your internal codebase can improve accuracy, but breaks when your team's coding patterns shift or you onboard new languages. Keep your fine-tuning data fresh or accept slightly lower task-specific accuracy from a general base model.

**Batch size=1 waste:** Single-request inference leaves GPU compute idle. But naively waiting 200ms to accumulate requests means the first developer in the batch experiences that full delay. For interactive code completion, p95 latency above ~150ms feels sluggish. The sweet spot often lands around 50–100ms batching windows.

Compile-first workflows help: Goatfied's agent loop runs lint/type checks before invoking the model, filtering out 40–60% of invalid edits. This reduces GPU load and surfaces errors faster than generate-then-validate patterns. Without validation, high-inference-throughput just means generating wrong code faster. The agent loop's constraint phase is architecturally cheap (it's a linter call, not a model forward pass) and saves expensive retry cycles.

Request routing and metrics that matter

A single GPU instance can't absorb traffic variance without either overprovisioning (wasting idle capacity) or queueing requests (blowing latency). You need a **request router** that tracks per-instance load and directs new requests to the least-busy GPU. NGINX with custom Lua scripts works; so does a simple Python FastAPI proxy that polls inference servers for queue depth.

For larger teams, consider multi-tenant setups: run multiple model replicas behind a load balancer, or use model multiplexing (loading several smaller models on one GPU and routing requests by task type). A 48GB L40S can fit a 7B completion model and a 13B refactoring model simultaneously, serving different endpoints. This increases complexity but improves cost efficiency when different workflows have different latency/quality tradeoffs.

Metrics that matter: p50/p95/p99 latency, requests per second, VRAM utilization, KV cache hit rate, token throughput. If p95 > 500ms, users notice. If VRAM util > 90%, you're about to OOM. Prometheus + Grafana with alerts on latency SLO violations. Regression in code quality is subtle—bad suggestions don't crash, they just frustrate users. Pair quantitative metrics (acceptance rate, tokens generated) with qualitative feedback from a few trusted devs.

**Fallback:** Keep API keys for a managed provider (OpenAI, Anthropic) as a hot standby. If your inference cluster is down or saturated, route overflow to the API. Costs spike but the tool stays usable.

When to stay managed

Self-hosting makes sense when:

• You're processing >10M tokens/day and API costs exceed $500/month
• Compliance requires data never leave your VPC
• You're fine-tuning models frequently and need tight training/inference integration
• Your usage is predictably high (>50% utilization over 24 hours)

It doesn't make sense when:

• Your team is <20 engineers with spiky usage
• You don't have ML infra expertise in-house (debugging CUDA OOMs and batching contention is not trivial)
• Your latency requirements are <200ms (self-hosted inference rarely beats well-optimized API endpoints on cold requests)

If you're under 50 engineers, have standard compliance needs, and don't need custom models, managed offerings beat self-hosting on total cost of ownership. The inflection point is around 100–150 seats, but it depends heavily on usage intensity and how much you value control vs. convenience.

Goatfied offers both paths: self-hosted deployments on your Kubernetes cluster (bring your own GPU nodes, we handle orchestration) or managed cloud instances where we absorb the infrastructure cost and you pay per seat. The self-hosted option makes sense for enterprises with existing GPU capacity or strict data residency rules (the agent loop's plan → constrain → edit → validate → retry cycle runs identically on your infra or ours). For smaller teams, the managed tier avoids the operational overhead of capacity planning, autoscaling, and model updates.

If you do self-host, start small. Rent a single L40S or A10G instance for a month, instrument everything (request latency, GPU utilization, token throughput, cost per 1k tokens), and compare against your current API bill. Only scale out once you've proven the unit economics work for your actual usage patterns. Self-hosting shines when you need reproducibility (every PR uses the exact same model checkpoint), auditability (logs never leave your VPC), or domain-specific fine-tuning (your codebase uses obscure DSLs). It's infrastructure, not magic—treat it like you'd treat a database cluster.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)