Zero-downtime model upgrades for coding assistants

Swapping out the LLM behind your coding assistant sounds simple in theory—update a version number, restart the service, done. In practice, you'll quickly hit a familiar ops problem: the moment you cut over to the new model, every in-flight request dies, developers see cryptic 503s, and you've just burned 15 minutes of your team's focus. If the new model has higher latency or different tokenization, those failures can cascade into retry storms.

Zero-downtime model deployment isn't exotic Netflix-scale thinking. It's table stakes once your coding assistant moves from "nice-to-have demo" to "blocking the build if it's down." This post walks through the mechanics: how to run two model versions in parallel, route traffic safely during the transition, and validate the new model under real load before you commit.

The naive approach and why it fails

The simplest mental model treats model inference like any stateless HTTP service: spin up a new container with the updated weights, update your load balancer target, kill the old container. For most CRUD APIs this works fine. For LLM inference it breaks in three predictable ways.

First, model loading is slow. A 70B parameter model can take 30–90 seconds to load into GPU memory, especially if you're quantizing on startup. If you terminate the old instance before the new one is serving, you've created a hard outage window.

Second, LLM requests are long-lived. Streaming completions for a complex code refactor might run 10–20 seconds. Killing the instance mid-stream doesn't just drop the request—it leaves the client in an ambiguous state where partial output has already been rendered in the editor.

Third, model behavior isn't deterministic across versions. A new model might produce different token probabilities for identical prompts, meaning a validation gate that passed under the old model could fail under the new one. If you haven't pre-tested this on real traffic patterns, your first signal is production errors.

Parallel deployment with traffic shadowing

The robust pattern borrows from blue-green deployments but adapts for the unique constraints of LLM inference. You run both model versions simultaneously, route live traffic to the stable (blue) version, and shadow a percentage of requests to the new (green) version without blocking on its responses.

Here's a minimal example using a reverse proxy to split traffic:

upstream blue_model {

    server model-blue:8080;

}



upstream green_model {

    server model-green:8080;

}



server {

    location /v1/completions {

        proxy_pass http://blue_model;

        

        # Shadow 10% to green, don't wait for response

        if ($request_id ~* "[0-9]$") {

            mirror /shadow;

            mirror_request_body on;

        }

    }

    

    location /shadow {

        internal;

        proxy_pass http://green_model;

    }

}

Shadowing lets you measure green's latency, error rate, and resource usage under production load without impacting users. The critical metric isn't just p50 latency—watch p99 and max resident memory. A new model that's 10% faster on average but spikes to 3× memory on complex prompts will OOM your pods during the next big refactor.

Validating output quality before cutover

Latency and uptime are easy to instrument. Output quality is harder because "correct" code isn't a boolean. You can't diff two completions and call one right—they might both compile, both pass tests, and have different design tradeoffs.

The approach that works: run the same validation gates you apply in your coding assistant's main loop, but compare pass rates between models. In Goatfied's agent architecture, every edit goes through compile → lint → test before it's accepted. Shadow traffic gives you a stream of (blue_output, green_output) pairs for the same prompt. Log whether each passes your gates.

// Simplified validation comparison

async function compareModelOutputs(prompt: string, blueCompletion: string, greenCompletion: string) {

  const [blueValid, greenValid] = await Promise.all([

    runValidationPipeline(blueCompletion),

    runValidationPipeline(greenCompletion)

  ]);

  

  metrics.increment('model.validation.blue', { passed: blueValid });

  metrics.increment('model.validation.green', { passed: greenValid });

  

  if (blueValid && !greenValid) {

    logger.warn('Green regression on prompt', { prompt, greenCompletion });

  }

}

If green's validation pass rate is within a few percentage points of blue over thousands of samples, you have reasonable confidence the quality delta is acceptable. If green passes 15% less often, you've caught a regression before users saw it.

Handling in-flight requests during cutover

Once shadowing shows green is stable, the actual cutover needs to drain blue gracefully. The pattern: stop sending new requests to blue, wait for existing requests to complete (with a timeout), then terminate blue's pods.

Kubernetes gives you this almost for free with graceful shutdown, but you need to tune the timeouts for LLM workloads:

apiVersion: apps/v1

kind: Deployment

metadata:

  name: model-blue

spec:

  template:

    spec:

      terminationGracePeriodSeconds: 60  # Long enough for slow completions

      containers:

      - name: inference

        lifecycle:

          preStop:

            exec:

              command: ["/bin/sh", "-c", "sleep 5 && kill -TERM 1"]

The preStop hook gives the process a chance to finish streaming responses before SIGTERM. Set terminationGracePeriodSeconds to your p99 completion time plus buffer—if 99% of requests finish in 30 seconds, 60 seconds is a safe margin.

On the client side, implement basic retry logic that doesn't assume the model URL is stable:

async function callModel(prompt: string, maxRetries = 2): Promise<Completion> {

  for (let i = 0; i < maxRetries; i++) {

    try {

      return await fetch('/v1/completions', {

        method: 'POST',

        body: JSON.stringify({ prompt }),

        signal: AbortSignal.timeout(45000)  // Fail faster than server timeout

      });

    } catch (err) {

      if (i === maxRetries - 1 || err.name !== 'AbortError') throw err;

      await sleep(1000 * Math.pow(2, i));  // Exponential backoff

    }

  }

}

The timeout on the client side should be shorter than the server's grace period. If a request is still running when the server starts shutting down, you want the client to give up and retry against a healthy instance rather than waiting 60 seconds for a response that may never come.

Model caching and warmup strategies

Cold starts are the silent killer of zero-downtime deployments. A freshly launched model instance might have empty KV caches, cold filesystem buffers, and unoptimized CUDA kernels. The first few requests can be 5–10× slower than steady state.

Warm up new instances before sending them production traffic:

# Warmup script run as init container or startup probe

def warmup_model(model_url: str, sample_prompts: list[str]):

    for prompt in sample_prompts:

        try:

            requests.post(f"{model_url}/v1/completions", 

                         json={"prompt": prompt, "max_tokens": 100},

                         timeout=30)

        except requests.Timeout:

            pass  # We're just warming caches, output doesn't matter

Pre-generate a handful of representative prompts—simple function implementations, test generation, refactors—and run them through the model before marking the pod as ready. This pre-fills the transformer KV cache and primes the GPU.

For self-hosted deployments with local model weights, consider mounting the model directory as a persistent volume that survives pod restarts. A 70B model is 140GB unquantized. Redownloading that on every deploy adds 10+ minutes to your rollout window.

Observability and rollback triggers

Deploy with clear rollback criteria defined before you start. Common signals that should trigger automatic rollback:

• Green error rate >2× blue's baseline for >5 minutes
• Green p99 latency >1.5× blue's baseline for >5 minutes
• Green memory usage exceeds pod limits (OOMKilled events)
• Green validation pass rate <90% of blue's rate over >1000 samples

Instrument both models with the same metrics and dashboards. When you're comparing two inference endpoints side-by-side, identical observability is non-negotiable:

// Middleware that works for both blue and green

function instrumentModel(modelName: string) {

  return async (req, res, next) => {

    const start = Date.now();

    res.on('finish', () => {

      const duration = Date.now() - start;

      metrics.histogram('model.latency', duration, { model: modelName });

      metrics.increment('model.requests', { model: modelName, status: res.statusCode });

    });

    next();

  };

}

Goatfied's managed platform handles this instrumentation automatically, but for self-hosted setups you need to build it yourself. Export metrics to Prometheus or Datadog and set up alerts that compare blue vs. green directly.

Related reading

• Self-hosted AI coding assistants: a complete Goatfied deployment guide
• Running AI coding assistants in air-gapped environments