Fine-tuning vs RAG for code assistants: a production playbook

Most teams building code assistants ask "should we fine-tune or use RAG?" and discover the real answer only after burning weeks on the wrong approach. The honest trade: fine-tuning bakes your patterns into model weights so you spend fewer tokens per request; RAG injects fresh context at inference time so your assistant doesn't suggest deprecated APIs. Production systems almost always need both, and the critical decisions are which context sources feed which pipeline stage, where you introduce validation gates, and how you handle the inevitable failures when the model halts mid-edit or generates plausible garbage.

Here's what actually matters: fine-tuning wins when your codebase has stable, opinionated conventions and you can afford weekly retraining. RAG wins when your code changes faster than you can fine-tune, or when correctness depends on reading files modified yesterday. The hybrid approach—fine-tune for style, retrieve for specificity, validate with compile/lint/test—is how you ship pull requests that pass CI on the first attempt instead of wasting engineer time on review corrections.

Why fine-tuning alone fails in production

Fine-tuning a model on your repository teaches it your naming conventions, error handling patterns, and framework idioms. It works beautifully for autocompletion when you're writing code that looks like code you've written before. The model "knows" that in your React components, API calls always live inside a `useQuery` hook with specific error boundaries, because it learned that pattern across hundreds of training examples.

The failure mode appears the moment your codebase changes—a refactor, a dependency upgrade, a new architectural decision. You're now serving suggestions based on code that no longer exists. One team fine-tuned on their Python backend weekly and still found the model suggesting deprecated ORM methods removed three weeks prior. The training run that took four days and cost significant GPU budget is obsolete the moment you merge a major refactor.

**The retraining cadence becomes a product problem.** Daily fine-tuning is expensive and slow; weekly runs leave your assistant perpetually behind. Fine-tuning also struggles with cross-repository reasoning. If your assistant needs to understand how a shared library is used across five services, you either fine-tune on all five (expensive, potentially leaking context across boundaries) or accept hallucinated usage patterns.

What fine-tuning actually learns is your codebase's chronological accident report, not a curated best-practices library. That 80,000-line monolith from 2019 is still the bulk of your training data, teaching the model antipatterns you're actively trying to eliminate. If your repo has inconsistent error handling—some functions return `Result<T>`, others throw exceptions, others return null-laden tuples—the fine-tuned model generates all three approaches in the same file. It's optimized for "looks like our code," not "follows our current standards."

Where RAG breaks down in code retrieval

RAG keeps the model frozen and solves the staleness problem: you index your codebase, and at inference time you retrieve the 10–20 most relevant chunks to stuff into context. When an engineer asks "how do we handle rate limiting?", you pull the actual rate-limiter implementation and recent usage examples. The assistant surfaces changes merged yesterday without retraining.

The breakdown happens in three predictable places. **Chunking destroys semantic boundaries.** Splitting a 2,000-line service file into 512-token windows means your retriever returns half a class definition or an import block with no function body. The model halts because it's missing the method signature it needs. Overlap helps but doubles your token spend and still misses cross-file dependencies.

**Embedding models don't understand call graphs.** Code isn't prose—semantic similarity between docstrings doesn't mean two functions are related in practice. If you're editing `handler.ts` and it imports a helper from `utils/validation.ts`, lexical similarity won't reliably surface that dependency. You retrieve a utility function because it has similar variable names to the one the engineer actually needs, or you miss a critical callback passed three layers deep. You need explicit graph traversal (AST-based or LSP-driven) to fetch transitive imports, which means your retrieval step is no longer a single vector search—it's a multi-hop pipeline with its own failure modes.

**Context windows fill up fast.** Even with 128k-token models, a typical feature touches 8–12 files. Add test fixtures, type definitions, and README snippets and you've burned 40,000 tokens before the model writes a single line. You need ranking heuristics—recency, edit frequency, test coverage—to prune aggressively, but every pruned file is a potential hallucination trigger.

Latency matters more than expected. A 200ms retrieval delay feels instant in a chat interface but is unacceptable for inline autocomplete. You need tiered systems: fast local index for autocomplete, slower but comprehensive vector search for explicit queries.

The hybrid pattern that actually works in production

Treat fine-tuning as your base layer for style and common patterns, then use RAG to inject fresh context for anything that touches recently modified files or involves cross-file dependencies your static analysis can trace. Gate all outputs through compile/lint/test before they reach a human.

Here's a concrete architecture that balances RAG flexibility and fine-tuned efficiency:

**1. Plan stage (RAG-driven context).** Engineer describes the task. Retrieve relevant files via AST traversal plus vector search. Generate a plan: "edit `routes.ts` to add endpoint, update `tests/api.test.ts`, add type to `types/index.ts`." Goatfied's retrieval layer treats code as a graph, not a bag of chunks—when the agent plans an edit, it walks the import tree and surfaces only files the compiler would need to validate the change. This keeps context grounded in actual dependencies, not embedding cosine scores.

**2. Constrain stage (fine-tuned adapter or prompt library).** Fetch exact current content of target files. If you've fine-tuned a small adapter (7B–13B parameters) on your team's code style and common patterns using LoRA or similar parameter-efficient methods, use it to propose diffs. This teaches variable naming conventions, error handling shapes, logger call signatures—stable patterns that don't change weekly. Otherwise, use a few-shot prompt with examples from your style guide.

**3. Edit stage (small reversible diffs).** Apply changes one file at a time, keeping each diff under 50 lines. Goatfied targets single-function or single-class changes. Larger refactors break into multiple steps. This constraint makes failures easy to diagnose and revert, and forces the model to reason incrementally instead of attempting full-file rewrites.

**4. Validate stage (compile/lint/test gates).** Run the same checks you'd run in CI, but fail fast—10 seconds max. TypeScript `tsc`, Rust `cargo check`, ESLint, clippy, fast unit tests. If checks fail, feed the error message back to the agent and retry (max 3 attempts). If all retries fail, surface the error to the user with the partial diff and logs. This closed-loop validation is how you produce diffs that pass CI on the first attempt. The model learns—within a single session—that it forgot an import or passed the wrong argument type.

The fine-tuned model gives you fast, stylistically consistent completions without burning 2,000 tokens per request on prompt engineering. RAG injects fresh context only when needed—recent refactors, new dependencies, current file state. The validation layer rejects anything that would break before it reaches code review. We've seen this pattern cut "looks right but doesn't compile" failures by roughly 80% compared to fine-tuning alone.

# Simplified pipeline structure

semantic_matches = vector_store.search(current_file_embedding, k=3)

symbol_defs = static_analyzer.find_definitions(imported_symbols)



prompt = f"""

Task: {edit_request}



Similar patterns from codebase:

{semantic_matches}



Current symbol definitions:

{symbol_defs}



Generate a diff that passes lint and tests.

"""



# Generate with fine-tuned adapter if available

diff = model.generate(prompt)



# Validate immediately

result = subprocess.run(["npm", "run", "lint"], capture_output=True, timeout=10)

if result.returncode != 0:

    retry_with_error(result.stderr)

When to bias toward fine-tuning

Fine-tuning wins when your codebase has stable, opinionated patterns—think a monorepo with strict lint rules and code generators—and you have infrastructure to retrain weekly or on-demand after major refactors. Latency is critical and you can't afford RAG's retrieval overhead for every keystroke.

For autocomplete specifically, a well-tuned model feels magical because it "just knows" your patterns. RAG struggles here—you can't wait 200ms to retrieve context before showing a suggestion. Fine-tune a small adapter on your diff style and code review comments, then use it as a proposal generator. Feed its output to a larger base model with RAG-augmented context for validation. The small model is cheap to retrain weekly; the large model stays grounded in real code.

**The practical constraint:** you need clean training data at scale. A few dozen examples won't move the needle. Fine-tuning pays off when you can extract hundreds of high-signal code pairs—every function that touched a specific module over the past year, filtered for files that passed review without comments.

When to bias toward RAG

RAG wins when your codebase changes rapidly, spans many repositories you can't easily train on together, or you need explainability—RAG lets you show the engineer exactly which files informed the suggestion. RAG also dominates for long-tail queries. Your codebase might have one esoteric PostgreSQL advisory lock pattern that appears in three files. Fine-tuning will underweight that rare signal; RAG will retrieve those exact three examples when the context matches.

If you're starting out and don't have ML infrastructure to retrain models regularly, RAG gets you to production faster. RAG also composes better with the agent loop. When Goatfied's planner decides to edit a file, it retrieves that file's recent git history, current tests, and dependent imports as part of the planning context. The model doesn't need to have "seen" those files during training—it just needs to reason about them in the moment.

**The quality bottleneck:** retrieval precision. You need code-specific embeddings trained on code search tasks, not off-the-shelf sentence transformers. We've seen teams spend weeks tuning their LLM while using generic embeddings for retrieval, then wonder why retrieved chunks are useless. Often you need a reranking pass to filter retrieved results before stuffing them into context.

Implementation details that matter

**Embedding models are as critical as your LLM.** Code-specific embeddings make a measurable difference. The gap between a model trained on natural language and one trained on code search tasks shows up as the difference between retrieving "functions with similar variable names" versus "functions that are actually imported by this file."

**Cache aggressively.** The same files get retrieved repeatedly. A 10-minute TTL cache on embeddings and retrieved chunks cuts costs by 60–70% without hurting freshness for most workflows. Self-hosted Goatfied users often extend this with Redis or in-memory caches keyed on file hashes.

**Index build times compound.** If your CI takes 5 minutes and your embedding index rebuild takes another 10, you've just added 15 minutes to every PR where an engineer wants an AI suggestion. Build incrementally: only re-embed changed files. Track file hashes and skip unchanged content. For large monorepos, this is the difference between "index rebuild completes in 30 seconds" and "index rebuild times out after 20 minutes."

**The constraint file becomes documentation.** Instead of "generate code and hope," define boundaries first: What files can change? What interfaces must remain stable? What tests must still pass? Then generate within those constraints. When validation fails, the constraint set tightens and the agent retries with error feedback.

goatfied agent \

  --task "add rate limiting to /api/search" \

  --preserve-interfaces "RateLimiter,SearchService" \

  --require-tests "test_rate_limit_enforcement" \

  --max-files 3

The agent can't hallucinate a new rate limiting implementation that violates your existing `RateLimiter` interface, because interface stability is a hard constraint. It can't skip tests, because test passage is a gate in the validation step. The diff size limit forces incremental changes that are easy to review and revert. The constraint file lives in version control, changes go through code review, and everyone—human and AI—works from the same rulebook.

The operational reality: cost and latency

Fine-tuning costs are upfront and predictable—one training run per week, storage for checkpoints. RAG costs scale with usage: every inference hits your vector DB and burns retrieval tokens. For high-volume agents making thousands of edits daily, RAG can quietly become your largest infrastructure line item.

Latency tells the opposite story. RAG adds 50–200ms for vector search and reranking before the model starts generating. Fine-tuned models respond immediately. If your code assistant is interactive—waiting for a developer to review suggestions—that latency is invisible. If you're running autonomous agents in a tight loop (plan, edit, validate, retry), milliseconds accumulate.

Goatfied's managed platform runs both: fine-tuning adapters update nightly in the control plane, while retrieval happens in-region with sub-100ms p95 latency. Self-hosted users often start with RAG-only because it's simpler to deploy, then add fine-tuning once they have usage data showing where the model repeatedly generates off-style code.

Making the call for your system

If your codebase is stable, your team has strong conventions, and you can invest in training infrastructure, fine-tuning delivers the cleanest output. If your codebase is fast-moving, your team is polyglot, or you need results this week, RAG gets you to production faster.

The real answer: most teams end up using both, with the boundary determined by what changes frequently (RAG) versus what should be consistent muscle memory (fine-tuning). Start with RAG because it's lower-risk and you'll learn what your model struggles with. Fine-tune an adapter once you've seen a thousand real prompts and know which patterns need to be internalized.

Code assistants that ship working PRs need both semantic understanding and immediate context. Neither approach alone survives contact with a production codebase where half the team just adopted a new pattern and the other half is still refactoring legacy code. Build the hybrid, gate with validation, and let the compile step catch errors before they reach code review. The constraint layer—not the model architecture—is what makes the system reliable enough to trust.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)