Fine-tuning vs RAG for code assistants: a production playbook

When building code assistants that actually ship features—not just autocomplete toys—you face a hard architectural choice: fine-tune a foundation model on your codebase, or layer retrieval-augmented generation (RAG) over it? Most engineering blogs hand-wave this decision with "it depends," then cite vanity metrics. This guide walks through the production tradeoffs, cost structures, and failure modes teams hit after the demo works.

Both approaches solve the same problem: a base language model trained on public GitHub repos doesn't know your API conventions, internal libraries, or the architectural decisions buried in three-year-old Slack threads. Fine-tuning bakes that knowledge into model weights; RAG fetches relevant snippets at inference time and stuffs them into context. The right choice hinges on update cadence, query latency, team size, and whether your codebase changes faster than you can retrain.

How fine-tuning works in practice

Fine-tuning takes a pre-trained model—GPT-4, CodeLlama, StarCoder—and continues training on your private dataset: pull requests, code reviews, documentation, resolved incidents. The model adjusts its weights to predict your patterns: naming conventions, error handling idioms, import hierarchies.

Preparation costs are front-loaded. You curate training data (removing secrets, normalizing file paths, balancing languages), spin up GPU clusters, and run distributed training jobs that can take hours to days. A team fine-tuning Llama 3.1 70B on ten thousand internal PRs might burn $2,000–$5,000 in compute per training run, plus weeks of iteration to tune hyperparameters.

The output is static. Once trained, the model's knowledge freezes. If a developer refactors your auth module next Tuesday, the assistant won't know until you retrain. Teams with fast-moving codebases—startups shipping daily—often find fine-tuned models stale within a sprint. Larger organizations on quarterly release cycles tolerate the lag.

Latency and hosting trade-offs. A fine-tuned model runs inference without external lookups, so generation is fast: 20–50 tokens per second on modest hardware. But you own the infrastructure: GPUs, model serving, autoscaling, monitoring. Open-source frameworks like vLLM and TensorRT-LLM ease deployment, yet you still need ML ops capacity. Hosted fine-tuning (OpenAI, Anthropic) offloads ops but locks you into their pricing and rate limits.

When fine-tuning wins: Long-lived codebases with stable architecture, compliance requirements that forbid sending code off-premises, or tasks where low latency (under 100ms to first token) justifies infrastructure overhead.

How RAG works for code assistants

RAG keeps the foundation model unchanged and injects relevant code snippets into every prompt. When a developer asks "How do I paginate this query?", the system retrieves examples from your repo, embeds them in the prompt, and the model generates an answer grounded in actual usage.

The retrieval pipeline is everything. You chunk source files (functions, classes, docstrings), generate embeddings with models like text-embedding-3-large or open-source alternatives (Nomic Embed, E5), and store them in a vector database (Pinecorn, Weaviate, Qdrant, pgvector). At query time, you embed the user's question, fetch top-k similar chunks, rerank them (Cohere Rerank, cross-encoders), and assemble the prompt.

Cost structure is per-query. Every request pays for embedding generation, vector search, and LLM tokens (both retrieved context and generated output). A RAG system answering 10,000 developer queries per day might spend $500–$2,000 monthly on embeddings and LLM calls, depending on context window size. No upfront GPU investment, but variable costs scale with usage.

Updates are near-real-time. When code changes, you re-index the affected files—often in seconds. Some teams hook RAG pipelines to CI/CD: every merged PR triggers incremental re-embedding. The assistant reflects the latest codebase state without retraining.

Failure modes are observable. When RAG hallucinates or misses an example, you inspect the retrieved chunks: Was the right code embedded? Did reranking fail? Did the prompt construction drop context? Debugging is forensic—logs, embeddings, similarity scores—rather than opaque gradient descent.

When RAG wins: Rapidly evolving codebases, teams without ML ops headcount, projects where explainability matters (you can cite retrieved examples), or when you want to pilot fast with managed APIs before committing to infrastructure.

Hybrid architectures: when to combine both

Production systems often blend fine-tuning and RAG. Fine-tune a smaller model (7B–13B parameters) on your coding style and common patterns, then use RAG to pull in fresh API changes or niche library usage. The fine-tuned base handles idioms; retrieval handles exceptions.

Example workflow: A fintech startup fine-tunes CodeLlama 13B on transaction processing logic (stable, security-critical), but RAG-augments queries about third-party integrations that change monthly. The model "knows" their architecture; RAG provides up-to-date SDK examples.

Complexity compounds. You maintain two systems: training pipelines and embedding infrastructure. Monitoring needs to distinguish model drift from retrieval failures. Most teams start with one approach, then hybridize only when clear performance gaps emerge—not as a first move.

Cost comparison: a worked example

Consider a 50-engineer team querying a code assistant 200 times per day (10,000 queries/month).

RAG estimate:

• Embedding: 10k queries × $0.0001/query (text-embedding-3-large) = $1
• Retrieval: pgvector on a $50/mo Postgres instance
• LLM inference: 10k queries × 4k context tokens × $0.01/1k tokens = $400
• Total: ~$450/month

Fine-tuning estimate:

• Training: $3,000 one-time, then $1,500/quarter for retraining
• Inference: GPU instance (g5.2xlarge, $1.20/hr) = $864/month
• Total: ~$1,364/month (amortizing training over three months)

RAG is cheaper below ~15,000 queries/month; fine-tuning becomes cost-effective at scale or when you need sub-50ms latency.

Evaluation beyond BLEU scores

Both approaches demand real-world metrics. Track:

• Acceptance rate: Percentage of generated code merged without edits
• Time to resolution: How quickly developers complete tasks with vs. without the assistant
• False confidence: Instances where the assistant sounds correct but suggests deprecated APIs or security anti-patterns
• Staleness lag: For fine-tuning, measure days between code change and assistant awareness; for RAG, track indexing latency

Anonymous A/B tests—half the team uses the assistant, half doesn't—reveal impact on velocity and bug introduction rates better than synthetic benchmarks.

Open-source tooling landscape

Open-source ecosystems for both paths have matured:

• Fine-tuning: Hugging Face PEFT (LoRA, QLoRA), Axolotl, LLaMA-Factory for efficient training; vLLM, TGI, Ollama for serving
• RAG: LangChain, LlamaIndex for orchestration; Chroma, Qdrant, Milvus for vector stores; Sentence Transformers for embeddings
• Hybrid frameworks: Haystack, Semantic Kernel support both retrieval and fine-tuned backends

Teams often prototype with these stacks, then migrate to managed services (Modal, Baseten, Anyscale) as query volume grows.

Platforms like Goatfied take a different tack: rather than building custom code assistants, teams hire AI agents from engineering roles—each pre-configured with decision rules and escalation paths—that work alongside the AI code editor and cloud coding agents to test changes and open pull requests. The integrated approach avoids stitching together fine-tuning pipelines and RAG layers yourself.

Frequently asked questions

Can I fine-tune on proprietary code without leaking IP?

Yes, if you control the infrastructure. Fine-tune open-source models (Llama, CodeLlama, StarCoder) on your own GPUs or use hosted fine-tuning services with contractual data isolation guarantees. Avoid fine-tuning APIs that incorporate your data into shared models unless the terms explicitly prevent it. Always audit training data for secrets (API keys, credentials) before feeding it to any model.

How often should I retrain a fine-tuned code assistant?

Depends on codebase churn. If fewer than 10% of files change per month, quarterly retraining suffices. High-velocity teams (daily deploys, microservices sprawl) retrain monthly or adopt RAG instead. Monitor staleness: if developers report outdated suggestions weekly, your cadence is too slow.

Does RAG work with large monorepos?

Yes, but chunking strategy matters. For multi-million-line repos, chunk by logical boundaries (classes, modules) rather than fixed token counts, and use hierarchical retrieval: first retrieve relevant packages, then zoom into functions. Precompute embeddings nightly; incremental updates handle daily commits. Teams with monorepos over 5M lines often shard the index by service or domain.

What's the minimum team size to justify building a custom code assistant?

RAG setups work for teams as small as ten engineers if you use managed APIs (OpenAI embeddings, Pinecone). Fine-tuning demands more: 50+ engineers to generate enough training data, plus ML ops capacity. Below that threshold, off-the-shelf assistants (GitHub Copilot, Cursor) or integrated platforms offer faster ROI than custom builds.

Can I switch from RAG to fine-tuning later?

Yes. Start with RAG to validate usefulness and gather query logs. Those logs become training data: user questions, retrieved snippets, and accepted vs. rejected suggestions. After six months, you'll have thousands of examples to fine-tune effectively. Many teams pilot with RAG, then fine-tune a smaller model for frequent patterns while keeping RAG for edge cases.

Related reading

• Why we open-sourced Goatfied (and what we kept proprietary)