Cost modelling for self-hosted inference at team scale

Running a large language model on your own infrastructure looks simple on paper: rent a server, load the weights, call the API. The reality is messier. A single NVIDIA A100 GPU costs around $3/hour on AWS or GCP, while inference traffic spikes without warning, models sit idle overnight, and quantization trades accuracy for savings you can't always measure. For a team of ten engineers, the monthly bill can range from $500 to over $5,000 depending on prompt volume, model size, and how aggressively you optimize—yet most cost calculators ignore utilization rates and the operational hours you'll spend tuning concurrency.

This guide walks through the real variables that drive self-hosted LLM costs at team scale: GPU selection, utilization patterns, prompt token throughput, quantization tradeoffs, and the hidden tax of maintenance. You'll learn how to model monthly spend, size infrastructure to match actual usage, and decide whether self-hosting beats managed API pricing for your workload.

GPU and compute costs: the baseline

Your largest line item is always the GPU. Popular choices for self-hosted inference:

| GPU model | VRAM | Cloud hourly rate (illustrative) | Suitable for |

|-----------|------|----------------------------------|--------------|

| NVIDIA A10 | 24 GB | ~$1.20 | 7B–13B models, single replica |

| NVIDIA L4 | 24 GB | ~$0.80 | 7B models with quantization |

| NVIDIA A100 40GB | 40 GB | ~$3.00 | 13B–34B models, higher concurrency |

| NVIDIA A100 80GB | 80 GB | ~$4.50 | 34B–70B models, lower quantization |

Self-hosting doesn't mean buying hardware—most teams rent cloud GPU instances. A 13B parameter model in FP16 needs roughly 26 GB of VRAM; add memory for the KV cache (which grows with context length and concurrent requests) and you're looking at an A100 40GB minimum. If you quantize to 4-bit, a single A10 or L4 may suffice, cutting hourly cost by 60–75%.

Utilization is where the math breaks. If your team's inference load spans 10 hours per business day and the instance runs 24/7, you're paying for 14 idle hours daily—about 58% waste. Auto-scaling helps, but cold-start latency (30–90 seconds to load model weights) forces you to keep at least one warm replica, which reintroduces baseline cost.

Example: A single A100 40GB running continuously is ~$2,160/month. If actual inference happens 50 hours/week, effective utilization is 30%, meaning you pay $7.20/productive hour instead of $3.

Prompt volume and token throughput

Monthly cost scales with how many tokens you generate. Measure your workload in:

• Input tokens per request: Typical code completion is 500–2,000 tokens; chat or document Q&A can be 4,000+.
• Output tokens per request: Code generation may produce 100–500 tokens; summarization 200–1,000.
• Requests per day: A team of 10 engineers making 50 requests each is 500 daily, or 10,000/month.

A 13B model on an A100 can generate 40–80 tokens/second depending on batch size and sequence length. If you process 10,000 requests/month, each averaging 1,500 input + 300 output tokens, that's 18 million tokens total. At 60 tokens/second, you need ~83 GPU-hours/month—well within a single always-on instance, but still 11% of the 730-hour month.

Batch inference vs. real-time: If requests can queue for seconds, batching four concurrent requests often doubles throughput per GPU-hour. Real-time code completion demands sub-200ms P95 latency, forcing smaller batches and lower utilization.

Quantization and model size tradeoffs

Quantization compresses model weights from 16-bit floats to 8-bit or 4-bit integers, halving or quartering memory use and often speeding inference. The cost benefit is real:

• FP16 (no quantization): Full quality, maximum VRAM and compute.
• 8-bit (int8): ~50% memory savings, negligible quality loss for many tasks, requires Ampere or newer GPUs.
• 4-bit (GPTQ, AWQ): ~75% memory savings, 2–5% quality degradation on benchmarks, faster on recent architectures.

Smaller models cost less to host but may require more requests to achieve the same result quality. If a 7B model needs two tries where a 34B model succeeds once, you double token throughput and potentially negate the GPU savings. Benchmark on your actual use case: coding tasks, for instance, often see meaningful accuracy drops below 13B parameters.

Load balancing and replica scaling

A single GPU instance becomes a bottleneck when concurrent requests exceed its batch capacity. Horizontal scaling—adding replicas behind a load balancer—improves throughput but multiplies cost linearly.

Common patterns:

• Single replica, autoscaling off-hours: Shut down nights and weekends, saving ~60% on a 40-hour workweek. Cold-start penalty acceptable for internal tools.
• Two replicas, round-robin: Handles peak traffic, provides fault tolerance. Doubles GPU spend.
• Burstable scaling: One warm replica, second spins up when queue depth exceeds a threshold. Balances cost and latency.

Most teams start with a single always-on instance. Add a second replica only when P95 latency breaks SLAs or request queues regularly exceed 10 seconds.

Operational overhead: the hidden tax

Self-hosting means owning the stack: model serving (vLLM, TGI, Ollama), monitoring, updates, and security patches. Budget for:

• Engineering time: 4–8 hours/month for a stable setup; 20+ during initial tuning or model upgrades.
• Storage: Model weights (13B ≈ 26 GB), logs, and occasional A/B tests. S3 or block storage adds $5–20/month.
• Networking: Egress fees if inference clients are outside your VPC. Often negligible at team scale but can surprise you.
• Monitoring and observability: Prometheus, Grafana, or vendor tools. Open-source is free; managed services run $20–100/month.

If your loaded engineer cost is $100/hour, eight hours of maintenance is $800—often more than the monthly GPU bill for a small team. Platforms that bundle inference, monitoring, and workflows reduce this drag. For example, Goatfied's AI agents handle repetitive tasks across departments, letting engineers focus on model performance rather than infrastructure babysitting, though Goatfied itself runs as a managed service rather than a self-hosted deployment.

Worked example: 10-person team

Assume:

• 10 engineers, 50 requests/day each = 10,000 requests/month
• 1,200 input tokens, 250 output tokens average
• 13B model, 4-bit quantization on A10 GPU ($1.20/hour)
• Instance runs 12 hours/day, 22 business days/month = 264 hours

Compute: 264 hours × $1.20 = $317/month

Storage: 15 GB model + logs ≈ $3/month

Engineering: 6 hours/month × $100 = $600/month

Total: ~$920/month

Compare to a managed API at $0.40/million input tokens and $1.20/million output:

• (10,000 × 1,200) / 1,000,000 × $0.40 = $4.80 input
• (10,000 × 250) / 1,000,000 × $1.20 = $3.00 output
• Total API: ~$8/month

For this light load, managed APIs win. Self-hosting becomes competitive above ~500,000 tokens/day (15M/month) or when data residency, latency, or model customization justify the operational cost.

Rightsizing for your workload

Start by measuring:

1. Request distribution: Log a week of real traffic—timestamps, token counts, P95 latency.

2. Peak vs. average: If peak is 3× average, you either overprovision for rare spikes or accept occasional queuing.

3. Model quality floor: Benchmark 7B, 13B, and 34B models on ten representative tasks. Quantify accuracy vs. cost.

Run a two-week proof of concept on a single GPU instance with autoscaling. Track:

• Actual utilization percentage
• Queue depth and latency at peak
• Failed requests or out-of-memory errors
• Time spent on setup and tuning

If utilization stays below 40% and maintenance eats more than four hours/week, managed inference or a platform that handles infrastructure likely offers better unit economics.

When self-hosting makes sense

Self-hosting wins when:

• Volume justifies fixed cost: Above 20–50 million tokens/month, GPU rental undercuts per-token API pricing.
• Data cannot leave your VPC: Regulatory or security requirements block external APIs.
• Custom fine-tuned models: You've invested in domain-specific weights and want sub-50ms latency.
• Batch workloads: Overnight processing, where you can shut down instances off-hours and tolerate cold starts.

For teams using AI across multiple functions—engineering, support, sales—centralizing infrastructure under a unified platform reduces duplication. Goatfied's IT suite and engineering tools coordinate workflows without requiring separate self-hosted stacks, though model hosting itself remains a build-vs.-buy decision based on your token economics.

Frequently asked questions

What's the break-even point between self-hosted and managed APIs?

It depends on token volume and engineering cost. As an illustrative scenario, if managed APIs charge $1/million tokens combined and your self-hosted GPU costs $2,000/month all-in, you break even around 2 billion tokens/month. Below that, APIs are cheaper; above, self-hosting saves money if utilization stays high.

Can I mix self-hosted and managed inference?

Yes. Route high-volume, latency-tolerant requests to your own GPUs and overflow or niche tasks to managed APIs. This hybrid approach caps your fixed cost while preserving flexibility.

How do I handle traffic spikes without overprovisioning?

Set request queues with reasonable timeouts (5–10 seconds), use autoscaling for predictable daily patterns, and fail over to a managed API when the queue exceeds a threshold. This "safety valve" prevents user-facing latency while keeping baseline cost low.

What's the true cost of maintaining a self-hosted LLM?

Count GPU hours, storage, networking, monitoring tools, and engineer time. For a stable setup, expect 4–8 hours/month; during model changes or framework upgrades, 20+ hours. At a $100/hour loaded cost, that's $400–$2,000/month in labor, often exceeding the infrastructure spend for small teams.

Should I buy GPUs or rent cloud instances?

Renting offers flexibility and eliminates capital expense. Buying makes sense only if you'll run 24/7 for 18+ months and have on-site power, cooling, and staff. Most teams under 50 people rent.

Related reading

• Self-hosted AI coding assistants: a complete Goatfied deployment guide