vLLM coding assistant stack: self-hosting from scratch

You've decided to run your own coding assistant infrastructure. Maybe you need air-gapped deployment, want to tune latency-cost tradeoffs yourself, or simply refuse to send proprietary codebases through third-party APIs. vLLM gives you production-grade inference serving with PagedAttention memory management and continuous batching—but standing up the full stack from bare metal to working autocomplete takes more than pip install vllm.

This guide walks through the actual components: model selection and quantization, vLLM server configuration, the stateless orchestration layer that connects editor requests to inference, and the observability hooks you'll need when token throughput doesn't match your projections at 3am.

Why vLLM for coding assistants specifically

Most LLM serving frameworks optimize for chat workloads: high per-request latency tolerance, large context windows, and conversational multi-turn exchanges. Coding assistance has different constraints. Autocomplete demands sub-200ms p95 latency for short completions (10-50 tokens), while fill-in-the-middle requests pull context from multiple files and can justify 1-2 second waits if the completion quality is high enough.

vLLM's continuous batching and prefix caching shine here. When five developers are working in the same repository, the shared imports and common utility functions get cached at the KV-cache level, reducing redundant computation. PagedAttention means you can pack more concurrent requests into GPU memory without the rigid static batching that kills tail latencies.

The tradeoff: vLLM is tightly coupled to CUDA and requires careful capacity planning. If you under-provision GPUs, queue depths spike and autocomplete timeouts cascade. Over-provision and you're burning cloud costs on idle V100 hours.

Choosing and quantizing your model

Start with the model architecture. For coding tasks, you want something trained on code-heavy datasets—StarCoder2, CodeLlama, or DeepSeek Coder variants. Download counts and leaderboard rankings are lagging indicators; test models against your actual codebase's language mix.

# Example: downloading StarCoder2-15B from Hugging Face

huggingface-cli download bigcode/starcoder2-15b \

  --local-dir ./models/starcoder2-15b \

  --local-dir-use-symlinks False

Quantization is where you trade quality for throughput. AWQ (Activation-aware Weight Quantization) and GPTQ get you to 4-bit precision with minimal accuracy loss. For a 15B parameter model, this means fitting on a single A10G (24GB VRAM) instead of requiring dual A100s.

# Using AutoAWQ to quantize a model

from awq import AutoAWQForCausalLM

from transformers import AutoTokenizer



model_path = "./models/starcoder2-15b"

quant_path = "./models/starcoder2-15b-awq"



model = AutoAWQForCausalLM.from_pretrained(model_path)

tokenizer = AutoTokenizer.from_pretrained(model_path)



quant_config = {

    "zero_point": True,

    "q_group_size": 128,

    "w_bit": 4

}



model.quantize(tokenizer, quant_config=quant_config)

model.save_quantized(quant_path)

Run local benchmark loops with your team's recent PRs as test data. Measure completion acceptance rate (what % of suggestions get used) and syntax error rate before you commit to a quantization strategy.

Configuring the vLLM server

vLLM runs as a standalone server process that exposes an OpenAI-compatible /v1/completions endpoint. Your editor plugins hit this API; vLLM handles batching, scheduling, and GPU orchestration.

python -m vllm.entrypoints.openai.api_server \

  --model ./models/starcoder2-15b-awq \

  --quantization awq \

  --dtype half \

  --max-model-len 8192 \

  --gpu-memory-utilization 0.9 \

  --host 0.0.0.0 \

  --port 8000 \

  --enable-prefix-caching

Key flags explained:

• --max-model-len: Context window limit. Coding assistants rarely need >8k tokens per request, and lower limits let vLLM batch more aggressively.
• --gpu-memory-utilization: Fraction of VRAM allocated to KV cache. 0.9 is aggressive but works if you're not running other GPU workloads.
• --enable-prefix-caching: Cache attention states for repeated prefixes. Essential when multiple requests share imports or class definitions.

For multi-GPU setups, vLLM handles tensor parallelism automatically if you pass --tensor-parallel-size 2. But watch for PCIe bandwidth bottlenecks—NVLink interconnects make a measurable difference above 2 GPUs.

Building the orchestration layer

The vLLM server is stateless. You need middleware that handles authentication, rate limiting, context collection from the editor, and request normalization. In Goatfied's architecture, this orchestration layer sits between the IDE plugin and vLLM, enforcing constraints before inference even starts.

A minimal orchestration service in Go or Python should:

1. Validate request schemas: Check that temperature, max_tokens, and stop sequences are within safe bounds.

2. Inject repository context: Pull relevant files based on imports or recent edits. This is where you integrate with your LSP or file indexing system.

3. Apply guardrails: Block requests that exceed token budgets, filter out secrets from context, log prompts for audit trails.

4. Route to vLLM: Forward validated requests to the inference backend and stream responses back to the client.

# Simplified FastAPI orchestration example

from fastapi import FastAPI, HTTPException

from pydantic import BaseModel

import httpx



app = FastAPI()



class CompletionRequest(BaseModel):

    prompt: str

    max_tokens: int = 50

    temperature: float = 0.2



@app.post("/v1/completions")

async def complete(req: CompletionRequest):

    if req.max_tokens > 100:

        raise HTTPException(400, "max_tokens capped at 100 for autocomplete")

    

    # Add context injection, guardrails, logging here

    

    async with httpx.AsyncClient() as client:

        resp = await client.post(

            "http://vllm-server:8000/v1/completions",

            json={"prompt": req.prompt, "max_tokens": req.max_tokens}

        )

        return resp.json()

In production, this layer also handles retries, circuit breaking, and graceful degradation if vLLM becomes unavailable. Goatfied's agent loop (plan → constrain → edit → validate → retry) relies on these gates to prevent runaway token consumption when validation fails.

Infrastructure and scaling considerations

vLLM runs best on bare-metal or VM instances with direct GPU passthrough. Kubernetes deployments work but add scheduling complexity—GPU node pools need careful autoscaling policies to avoid cold-start latency when request volume spikes.

For self-hosted stacks, start with a single-node deployment:

• Compute: 1x A10G or 1x L4 instance (AWS g5.xlarge, GCP g2-standard-4)
• Disk: 100GB SSD for model weights and logging
• Network: 10Gbps link if you're serving multiple remote offices

Monitor three key metrics from the start:

1. Queue depth: vLLM exposes this via Prometheus metrics. Sustained queues >5 mean you're under-provisioned.

2. Time-to-first-token (TTFT): Should stay <100ms for autocomplete. Spikes indicate prefix cache misses or batching delays.

3. GPU utilization: Target 70-85%. Below 50% means you can serve more load; above 90% risks OOM errors.

When you need to scale horizontally, run multiple vLLM replicas behind a load balancer. Sticky sessions aren't necessary—prefix caching works per-instance, and most coding assistant requests are independent.

Observability and debugging inference issues

Your first production incident will involve "completions are slow but GPU usage is low." This usually points to CPU-bound tokenization or inefficient batching.

# Check vLLM's internal queue and scheduling stats

curl http://localhost:8000/metrics | grep vllm_

Look for vllm_num_requests_running and vllm_time_to_first_token_seconds. If TTFT is high but running requests are few, you're likely seeing cold-cache misses or model swap delays.

Enable detailed logging to capture prompt lengths and token counts:

python -m vllm.entrypoints.openai.api_server \

  --model ./models/starcoder2-15b-awq \

  --log-level debug \

  > vllm.log 2>&1

For inference quality issues (hallucinated imports, broken syntax), log the full prompt+completion pairs. Build a regression test suite from accepted completions to catch model degradation after upgrades or re-quantization.

Connecting editors and IDE plugins

Most modern IDE plugins (GitHub Copilot, Cody, Continue) support custom inference endpoints. Point them at your orchestration layer's URL and configure bearer token authentication.

For VS Code with Continue:

{

  "models": [{

    "title": "Internal vLLM",

    "provider": "openai",

    "model": "starcoder2-15b-awq",

    "apiBase": "https://your-orchestration-service.internal/v1",

    "apiKey": "your-auth-token"

  }]

}

Test autocomplete latency from developer workstations, not the server itself. Network round-trip adds 20-50ms in typical office environments, and WiFi jitter can double p95 latencies.

Cost modeling and capacity planning

A single A10G instance costs roughly $0.50/hour on AWS (g5.xlarge spot). If you're serving 20 developers with average 30 requests/hour each, that's 600 requests/hour. At 30 tokens/request and 100ms TTFT + 20ms/token generation, you can handle ~200 concurrent requests/second per GPU.

This leaves headroom for burst traffic (post-standup coding sessions, CI/CD integration runs). For 50+ developers or lower latency requirements, provision 2-3 replicas and load balance by request count, not GPU utilization.

Compare this to API pricing: OpenAI's API charges $0.002/1k tokens. At 30 tokens/completion × 600 requests/hour × 730 hours/month, you'd spend ~$26/month per developer. Break-even happens around 15-20 developers for a single self-hosted instance, assuming you already have GPU infrastructure.

Related reading

• Self-hosted AI coding assistants: a complete Goatfied deployment guide
• Running AI coding assistants in air-gapped environments