Fine Tuning Vs Rag For Code Assistants Production: practical systems guide

Most teams building code assistants start by asking "should we fine-tune or use RAG?" This is the wrong question. Production systems almost always need both, and the real decision is which context sources feed which stage of your pipeline, and where you introduce fallback paths when your model halts or produces garbage.

Here's what actually matters: RAG gives you runtime control over what the model sees. Fine-tuning bakes patterns into weights so you spend fewer tokens on prompt engineering. You pick based on how often your codebase changes, how much latency you can tolerate, and whether your retrieval can stay accurate as repos grow. Most importantly, you need escape hatches for when the LLM fails mid-edit—this is where compile/lint gates and reversible diffs become load-bearing.

When RAG breaks down in code retrieval

RAG for code looks simple: embed files, retrieve top-k chunks, stuff them into context. In practice, you hit three walls fast.

**Chunking destroys semantic boundaries.** Splitting a 2,000-line service file into 512-token windows means your retriever hands back half a class definition or an import block with no function body. The model halts because it's missing the method signature it needs to complete the edit. Overlap helps but doubles your token spend and still misses cross-file dependencies.

**Embedding models don't understand call graphs.** If you're editing `handler.ts` and it imports a helper from `utils/validation.ts`, lexical similarity won't reliably surface that dependency. You need explicit graph traversal (AST-based or LSP-driven) to fetch transitive imports. This means your retrieval step is no longer a single vector search—it's a multi-hop pipeline with its own failure modes.

**Context windows fill up fast.** Even with 128k-token models, a typical feature touches 8–12 files. Add test fixtures, type definitions, and README snippets and you've burned 40k tokens before the model writes a single line. You need ranking heuristics (recency, edit frequency, test coverage) to prune aggressively, but every pruned file is a potential hallucination trigger.

Goatfied's retrieval layer solves this by treating code as a graph, not a bag of chunks. When the agent plans an edit, it walks the import tree and surfaces only the files the compiler would need to validate the change. This keeps context tight and grounded in actual dependencies, not embedding cosine scores.

Where fine-tuning actually helps (and where it doesn't)

Fine-tuning makes sense when you have repeated patterns the base model doesn't know. Examples: your team's custom test harness syntax, a DSL for infrastructure templates, or the specific sequencing of CLI commands in your CI scripts. Baking these into weights means you don't burn 2k tokens per request explaining how `goatfied deploy --env prod` differs from a standard docker push.

**What fine-tuning won't fix:** mistakes in reasoning. If your model generates a SQL query with a missing JOIN clause, fine-tuning on more examples of correct queries might help, but it won't teach the model to validate its output against a schema. You still need a validation step that runs `EXPLAIN` or a dry-run migration before committing.

**The update problem is real.** A fine-tuned model is frozen at training time. When you add a new API endpoint or refactor your auth middleware, the model doesn't know. You're back to RAG anyway to inject fresh context. Hybrid approaches work: fine-tune on stable patterns (test structure, PR conventions), then use RAG to surface the current state of `main`.

One tactic we've seen work: fine-tune a small model (7B or 13B) on your team's diff style and code review comments, then use it as a *proposal generator*. Feed its output to a larger base model with RAG-augmented context for validation. The small model is cheap to retrain weekly; the large model stays grounded in real code.

The validation layer no one talks about

Both RAG and fine-tuning produce drafts. The draft is not the deliverable. You need automated gates that catch errors before a human wastes time reviewing broken code. This is the least glamorous part of the stack and the most critical.

At minimum: compile checks (TypeScript `tsc`, Rust `cargo check`), linters (ESLint, clippy), and fast unit tests. Goatfied's agent loop runs these after every edit and feeds errors back into the next retry. The model learns—within a single session—that it forgot an import or passed the wrong argument type. This closed-loop validation is why we can produce small, reversible diffs instead of rewriting entire modules.

Contrast this with tools that dump a 300-line diff and hope the developer catches the mistake. If your system can't retry with compiler feedback, you're asking humans to be the error-correction layer. That's expensive and doesn't scale.

**Practical setup:** hook into your existing CI pipeline. Run the same checks locally that you'd run in CI, but fail *fast*—10 seconds max. If a check hangs, kill it and fall back to a simpler heuristic (syntax-only validation, or even a pattern match for common mistakes). The goal is to block obvious errors without becoming a bottleneck.

Putting it together: a hybrid pipeline

Here's a minimal production architecture that balances RAG flexibility and fine-tuned efficiency:

1. **Plan stage (RAG).** User describes the task. Retrieve relevant files via AST + vector search. Generate a plan: "edit `routes.ts` to add endpoint, update `tests/api.test.ts`, add type to `types/index.ts`."

2. **Constrain stage (fine-tuned model or prompt library).** Fetch exact current content of those files. If you've fine-tuned on your team's patterns, use that model to propose diffs. Otherwise, use a few-shot prompt with examples from your style guide.

3. **Edit stage (small diffs).** Apply changes one file at a time. Keep each diff under 50 lines. Larger refactors break into multiple steps.

4. **Validate stage (compile/lint/test).** Run checks. If they fail, feed the error back and retry (max 3 attempts). If all retries fail, surface the error to the user with the partial diff and logs.

5. **Audit stage.** Log the full trace: what was retrieved, what was generated, what passed/failed. Self-hosted Goatfied lets you keep these logs in your own infra; the managed version encrypts them at rest.

This pipeline is boring. It works. It doesn't rely on a magic model that "just understands" your codebase. It combines retrieval for freshness, optional fine-tuning for efficiency, and validation to catch errors before they leave the system.

Related reading

• [Why we open-sourced Goatfied (and what we kept proprietary)](/blog/why-we-open-sourced-goatfied-and-what-we-kept-proprietary)
• [Open-Source playbook #3: practical Goatfied tactics for shipping PRs](/blog/goatfied-open-source-playbook-3)