Running AI coding assistants in air-gapped environments

Organizations in regulated industries, defense contractors, and enterprises with strict data sovereignty requirements face a paradox: they need modern AI-powered development tools, but their networks are isolated from the internet by design. Air-gapped environments—systems physically or logically separated from external networks—traditionally mean giving up cloud-based tooling. But as AI coding assistants become infrastructure rather than luxury, teams are finding ways to run these capabilities behind the wall.

The challenge isn't just downloading a model file. A production-grade air-gapped AI coding assistant requires local inference infrastructure, model versioning, monitoring that works without phoning home, and integration with your existing development workflow—all while maintaining the compile-first reliability and auditability that likely motivated the air gap in the first place.

What actually needs to be isolated

Air-gapped doesn't always mean zero connectivity. Understanding your specific isolation requirements shapes the entire deployment:

Hard air gaps involve physical network separation. Code never leaves the facility. Model weights arrive on encrypted drives. Updates happen through formal transfer procedures with security reviews. This is common in classified government work, certain financial trading floors, and critical infrastructure.

Logical air gaps use strict network controls, allowlists, and proxy layers. The environment can't initiate outbound connections, but approved inbound traffic may be permitted through heavily monitored channels. This describes many healthcare systems, regulated pharma R&D environments, and defense contractors working on sensitive-but-unclassified projects.

Data residency constraints technically allow internet access but prohibit certain data from leaving geographic or organizational boundaries. Your code and proprietary context can't touch external APIs, but the environment itself isn't fully isolated.

Each scenario has different implications for model deployment, telemetry, and updates. A hard air gap means you're running your own model servers on-premises. Logical air gaps might allow pulling container images from approved registries. Data residency requirements might work with a self-hosted deployment that never sends code externally but can still ping update servers for the platform itself.

Model hosting inside the perimeter

The foundation is running inference locally. For transformer-based code models, this means:

Model selection trades capability for hardware constraints. A 7B parameter model like CodeLlama or StarCoder fits comfortably on a single GPU with 24GB VRAM. Larger models (30B+) deliver better results but require multi-GPU setups or careful quantization. In air-gapped contexts, you're balancing model quality against the hardware you can physically install and power.

Quantization becomes mandatory, not optional. Running int8 or even int4 quantized versions reduces memory footprint dramatically—a 13B model that normally needs 52GB can run in under 16GB quantized. The quality degradation is noticeable but often acceptable when the alternative is no AI assistance at all.

Inference server choice matters for ops. Tools like vLLM, TensorRT-LLM, or llama.cpp provide the HTTP layer your coding assistant hits. In air-gapped setups, pick based on your team's operational strengths. If you have strong Kubernetes skills, run vLLM in pods with autoscaling. If you're primarily a VM shop, llama.cpp with a small FastAPI wrapper might be simpler to support long-term.

Goatfied's architecture separates the agent loop (plan, constrain, edit, validate, retry) from the model backend. You point it at your inference endpoint via environment config—no code changes needed to swap from OpenAI to your internal vLLM cluster. This boundary makes air-gapped deployments feasible: the edit/validation machinery runs the same whether your model is in the cloud or in your datacenter.

Getting models and updates inside

Model weights are multi-gigabyte artifacts. In cloud environments, you download them once. In air-gapped scenarios, every model update is a logistical event:

Initial seeding typically uses physical media for hard air gaps. Download model checkpoints on a connected workstation, transfer to encrypted drives, move them through your approved media transfer process. This takes days or weeks depending on your security protocols, so picking the right model upfront matters.

Incremental updates are harder than initial deployment. When a new model version drops, you're repeating the physical transfer process. Many teams simply don't update frequently—staying on a known-good model version for months. This argues for initially deploying a slightly older, more stable model rather than chasing the latest release.

Container registries inside the firewall solve part of the update problem for the platform itself. If you can run a local Harbor or Artifactory instance, you transfer container images once, then standard Kubernetes or Docker workflows take over. Goatfied's self-hosted deployment is distributed as container images, so this pattern works: load the images into your internal registry, then your air-gapped clusters pull from there.

One pattern that works: maintain a separate "transfer zone" environment with limited internet access. Download models, scan them thoroughly, test in isolation, then formally promote to the air-gapped production environment. This reduces the iteration time for testing new models without compromising the production boundary.

The validation pipeline is your safety net

Without external API calls to catch errors, local validation becomes critical. This is where compile-first reliability matters more than in connected environments:

Lint and type checking must run immediately. If the AI suggests code that doesn't compile, you want to know before a developer commits it—because in an air-gapped environment, there's no "quick fix" from an external model. Goatfied runs golangci-lint, TypeScript's tsc, or similar checkers after every edit step, rejecting diffs that break compilation before they reach your review.

Tests become the ground truth for correctness. The validation phase in the agent loop runs your existing test suite against proposed changes. In air-gapped deployments, these tests are often the only automated signal that code works—no cloud CI to fall back on, no external SaaS to verify builds. Invest in comprehensive test coverage before deploying AI assistance.

Small, reversible diffs reduce risk. Goatfied's architecture generates minimal changesets that solve one thing at a time. In air-gapped contexts where debugging is harder (no quick external searches, no ChatGPT to explain errors), this constraint prevents the model from creating large, tangled changes that fail in subtle ways.

A financial services team running Goatfied in a logically air-gapped environment described their workflow: the AI generates a diff, local compilation validates syntax, unit tests confirm behavior, then security scans run before anything hits version control. Each gate catches issues that would be expensive to fix after the fact, when the model isn't available to quickly iterate.

Operational monitoring without telemetry

Standard SaaS tools send usage data, error reports, and performance metrics to external dashboards. Air-gapped deployments need observability that stays inside:

Local metrics collection replaces cloud telemetry. Run Prometheus inside your environment, scrape your inference servers and Goatfied instances, visualize in a local Grafana. Track inference latency, token throughput, validation failure rates, and edit acceptance ratios.

Log aggregation stays on-premises. Use ELK stack, Loki, or similar tools to centralize logs from all components. When something breaks—model quality degrades, validation times spike, agent loops timeout—you need to debug without opening a support ticket that requires sharing logs externally.

Audit trails for compliance. Air-gapped environments often exist because of regulatory requirements. Every code change, model invocation, and validation run should be logged immutably. Goatfied's architecture already tracks the full chain from plan to validated diff; in self-hosted deployments, those logs live in your data stores, queryable for compliance audits.

One practical detail: reserve capacity for observability infrastructure. In cloud deployments, you can scale monitoring on demand. In air-gapped setups, you're pre-provisioning resources. Budget for logging storage and metrics retention from day one.

Managing the context boundary

AI coding assistants need context—your codebase, internal libraries, API documentation. In air-gapped environments, this context can't be augmented with external knowledge:

Embeddings for codebase search must be generated locally. Tools that build vector indexes of your code for retrieval-augmented generation need to run on-premises. This means deploying an embedding model alongside your main code model. Smaller embedding models (like all-MiniLM-L6-v2) run cheaply on CPU.

Internal documentation becomes the knowledge base. Without access to GitHub discussions, Stack Overflow, or external docs, your internal wiki and API references are the only source of truth for library usage and patterns. Make sure these are indexed and accessible to your assistant's context-building phase.

Dependency resolution needs internal mirrors. If the AI suggests adding a library, you can't just npm install or go get from public registries. Maintain internal mirrors of approved dependencies (Artifactory, Nexus, or similar). Configure your assistant's environment to only reference these internal sources.

Weighing the tradeoffs honestly

Air-gapped AI assistance is possible but comes with real costs:

You'll lag external capabilities. The latest models arrive months late, if at all. Fine-tuning on proprietary code requires local ML expertise and GPU resources. Developer velocity gains are real but smaller than cloud-assisted teams see.

Operational overhead is significant. Someone needs to manage model updates, inference infrastructure, and troubleshoot local deployment issues without vendor support. Budget for dedicated DevOps or SRE capacity.

But for teams where air gaps are non-negotiable, having 70% of the capability of cloud assistants beats having 0%. The key is treating AI assistance as infrastructure: plan for updates, monitor carefully, and design your workflow around validation gates that catch issues the model can't fix externally.

Related reading

• Self-hosted AI coding assistants: a complete Goatfied deployment guide