Cost modelling for self-hosted inference at team scale

Learn how to calculate actual costs for self-hosted LLM inference by modeling compute utilization, instance sizing, and traffic patterns for small engineering teams.

Every team running self-hosted LLMs eventually asks the same question: are we actually saving money, or just trading OpenAI's invoice for a cloud compute bill that scales worse? The answer depends entirely on your traffic patterns, hardware choices, and how you handle the difference between peak and average load.

A typical 20-engineer team generating 500–800 completions per day might pay $400–600/month to OpenAI or Anthropic. Switching to self-hosted inference can drop that to $150–250 in compute costs, but only if you size correctly and accept that your model selection narrows. Get it wrong—overprovision by 2x or run too small a model—and you'll spend more while delivering a worse experience.

The three variables that determine total cost

Self-hosted LLM costs break down into compute, storage, and egress. Compute dominates: a single A10G instance on AWS runs roughly $1.50/hour ($1,100/month if reserved). Storage for model weights is negligible—even 70B parameter models fit in under 200GB. Egress matters only if you're serving traffic across regions or pulling weights repeatedly; most teams pay under $20/month here.

The real cost driver is utilization. If your team generates 600 completions daily with an average of 800 input tokens and 400 output tokens, you're processing roughly 720,000 tokens/day. A well-tuned Llama 3 8B instance on an A10G can handle 25–30 tokens/second, meaning that workload requires about 8 hours of full GPU time per day. Run that instance 24/7 and you're paying for 16 hours of idle capacity every day.

Rightsizing means either:

Accepting occasional queueing during peak hours (2–5pm) and running a single instance
Spinning up a second instance dynamically during known busy windows
Using a managed autoscaling layer that provisions capacity on-demand

Most teams start with option one. If your engineers tolerate 3–5 second response times during lunch-hour spikes, a single reserved instance with a small queue is the most cost-effective choice.

Model selection changes the math completely

Switching from GPT-4 to self-hosted isn't apples-to-apples. You're trading model capability for cost control, and the gap matters more in some workflows than others.

For code completion and inline suggestions, 8B parameter models (Llama 3.1 8B, CodeLlama 7B, DeepSeek Coder 6.7B) deliver acceptable results at ~30 tokens/sec on a single A10G. For architectural questions, refactoring, or complex debugging, you'll want 30B+ parameters—which means either:

A100 instances at $4–5/hour (reserved pricing)
Multi-GPU setups with tensor parallelism
Accepting 8–12 tokens/sec throughput on quantized 70B models

Here's where cost modeling gets specific. If 80% of your team's requests are autocomplete (low-stakes, high-volume) and 20% are chat/refactor (high-stakes, low-volume), you can run:

One A10G instance for completions: ~$800/month reserved
Burst to a larger instance for chat, billed hourly: ~$150–300/month depending on usage

That's $950–1,100/month total, but you've preserved quality on the workflows that matter. Compare that to an all-70B approach on a single A100 at $3,000+/month (reserved), and the hybrid strategy is obvious.

Measuring actual utilization before you commit

The worst mistake is provisioning based on gut feel. Before you reserve instances, instrument your current API usage for two weeks. Track:

Requests per hour (bucketed by hour-of-day and day-of-week)
Tokens per request (separately for input/output)
P95 response time expectations from your team

If you're using Goatfied's managed cloud, this data is already in the usage dashboard. If you're planning a self-hosted migration, add a lightweight logging middleware to your API client that writes request metadata to CloudWatch or an S3 bucket.

Look for patterns:

Do you have a weekday-only workload that drops to near-zero on weekends?
Is there a 10am–3pm peak that's 3–4x your overnight baseline?
Are 90% of requests under 1,000 input tokens, with a long tail of 5,000+ token context windows?

A team with steady 24/5 usage should reserve instances. A team with spiky weekday peaks should use spot instances or a managed autoscaler that provisions on-demand. A team with massive variance (100 requests/day Monday, 1,500 requests/day Friday during sprint planning) needs a cost model that assumes you're paying for peak capacity most of the time, or you need to rethink workflows to spread load.

Handling compile/test gates without runaway costs

Goatfied's agent loop—plan, constrain, edit, validate, retry—means LLM calls happen at multiple stages, not just during the initial code generation. If your validation step runs cargo check or pytest and then asks the model to fix errors, you're potentially doubling your token throughput during active development sessions.

This is where batching and priority queues matter. Not every agent retry needs subsecond latency. If an engineer submits a change and Goatfied's validator catches a type error, the model can take 8–10 seconds to generate a fix without degrading the experience—because the human is already context-switched to the test output.

Structure your inference server to:

Route interactive completions (typing in the editor) to a fast-path queue with dedicated capacity
Route validation-triggered fixes to a slower queue that batches requests when possible
Set a TTL on retries so you're not burning tokens on stale context after the engineer has moved on

This two-tier approach lets you run a smaller primary instance (optimized for p50 latency) and a secondary batch-oriented instance (optimized for throughput). Total cost stays flat, but perceived performance improves because the workloads that demand speed get it.

The hidden cost of model updates and experimentation

Self-hosted inference gives you control, but it also gives you maintenance burden. OpenAI silently upgrades GPT-4 and you get improvements for free. Self-hosted means you're manually pulling new Llama releases, re-quantizing weights, benchmarking performance, and rolling out updates.

Budget 4–8 hours per quarter for model evaluation. You'll want to:

Download new releases (DeepSeek, Qwen, Llama, CodeGemma, etc.)
Run a standardized eval suite on your team's actual codebase (not HumanEval)
Compare output quality and tokens/sec on your target hardware
A/B test with a subset of your team before rolling out broadly

This work is free in the sense that it's your engineers' time, not cloud spend—but it's not zero-cost. If you're a 10-person startup with one infra-focused engineer, the overhead might outweigh the API savings. If you're a 50-person team with dedicated platform ownership, it's tractable.

Goatfied's self-hosted deployment mode helps here by packaging model updates as versioned container images with pre-tuned inference configs. You still need to benchmark, but you skip the quantization and server-tuning steps.

When self-hosted costs more (and you should do it anyway)

There are cases where self-hosted inference is definitionally more expensive than API calls—and you choose it for non-cost reasons:

Air-gapped environments: If you're in defense, healthcare, or financial services with hard network boundaries, API calls aren't an option. You're comparing self-hosted to "no LLM-assisted tooling at all," and the ROI calculation is about developer productivity, not line-item cost.

Data residency requirements: GDPR, HIPAA, or contractual obligations that prevent sending code to third-party APIs mean self-hosted is the only compliant path. You might pay $2,000/month for inference that could cost $500 via API, but the alternative is legal/compliance risk you can't accept.

Auditability and determinism: Goatfied's compile-first workflow depends on being able to reproduce exact agent runs. If you need to prove that a given code change came from a specific model version with specific context, you need control over the inference stack. Managed APIs don't give you that.

In these scenarios, cost modeling still matters—you want to minimize waste—but the decision to self-host is already made. The question becomes "how do we do this efficiently" rather than "should we do this at all."