Fine Tuning Vs Rag For Code Assistants Failure: practical systems guide

Most teams discover their code assistant isn't working when they watch a senior engineer spend fifteen minutes correcting hallucinated imports, then abandon the tool entirely. The standard move is to throw more context at the problem—RAG over the codebase, fine-tuning on internal repos—but both approaches fail in predictable, expensive ways that nobody warns you about until you've already burned the budget.

The failure modes aren't what the vendor slides show. RAG systems return syntactically correct but architecturally wrong suggestions because vector similarity doesn't understand your team's unwritten rules about when to use Factory vs Builder patterns. Fine-tuned models confidently generate code in the style of your worst legacy module because training data doesn't distinguish between "code we're proud of" and "code we're paying down as technical debt." Both paths lead to the same place: engineers who stop trusting the assistant and revert to writing everything by hand.

Why retrieval fails at the boundaries

RAG works beautifully for documentation lookup and fails catastrophically for behavioral context. When an engineer asks "how do we handle rate limiting in this service," a well-indexed codebase returns the retry logic, the exponential backoff configuration, and the relevant middleware. That's table stakes.

The breakdown happens when the question is "should I add rate limiting to this new endpoint?" The retrieval system has no way to surface the Slack thread where the team decided rate limiting belongs in the API gateway layer, not individual services. It can't tell you that the VP of engineering vetoed per-endpoint limits after the Q3 incident. Vector embeddings capture "what exists," not "why we don't do X" or "the last three people who tried this approach and what went wrong."

In practice, this manifests as assistants that suggest plausible but policy-violating code. You retrieve examples of direct database access because those patterns exist in the codebase, even though the new standard is repository interfaces with explicit transaction boundaries. The RAG system doesn't know one is deprecated—it just knows both appear in recent commits.

Fine-tuning learns the wrong lessons

Training a model on your internal codebase sounds like the principled solution until you examine what the model actually learns. Code repositories are chronological accident reports, not curated best-practices libraries. That 80,000-line monolith from 2019? It's still the bulk of your training data, teaching the model patterns you're actively trying to eliminate.

We've seen this play out in two ways. First, the model learns your team's specific antipatterns with perfect fidelity. If your codebase has inconsistent error handling—some functions return `Result<T>`, others throw exceptions, others return null-laden tuples—the fine-tuned model will happily generate all three approaches in the same file. It's optimized for "looks like our code," not "follows our current standards."

Second, fine-tuning bakes in temporal context that expires immediately. Train on code written when your stack was Python 3.8 and Flask, and the model keeps generating Flask blueprints even after you've migrated to FastAPI. The training run that took four days and cost $$$ is obsolete the moment you merge a significant architectural change. You're now maintaining model retraining pipelines alongside your CI/CD, and that's before you've solved the "what is good training data" problem.

The hidden cost: validation tax

Both approaches push the verification burden onto humans, which defeats the entire point of automation. An engineer using a RAG-based assistant has to mentally compile every suggestion against unstated architectural principles. "This code looks right, but does it follow the new service mesh patterns?" becomes a constant low-grade cognitive load.

Fine-tuned assistants are worse because they generate code that *feels* native. The style matches, the naming conventions are right, but the actual logic implements a deprecated approach with perfect confidence. Engineers either learn to distrust the tool (and stop using it) or they let bad suggestions through because the code review process assumes human-written code defaults to trustworthy.

The real expense is the context-switching cost. Every suggestion becomes a research task: Is this the current pattern? When did we deprecate the old approach? What's the approved way now? You've replaced "write code" with "audit generated code against tribal knowledge," which is slower and more error-prone.

What actually works: constraint-first generation

The only reliable pattern we've found is inverting the problem: define the boundaries first, then generate within them. This is the core of Goatfied's agent loop—plan what changes are safe, constrain the edit space, make small modifications, then validate immediately with compile/lint/test gates.

Instead of "generate code and hope," the system asks: What files can change? What interfaces must remain stable? What tests must still pass? Then it makes the smallest diff that satisfies those constraints. When validation fails, the constraint set tightens and the agent retries with more information about what went wrong.

This looks like:

goatfied agent \

  --task "add rate limiting to /api/search" \

  --preserve-interfaces "RateLimiter,SearchService" \

  --require-tests "test_rate_limit_enforcement" \

  --max-files 3

The agent can't hallucinate a new rate limiting implementation that violates your existing `RateLimiter` interface, because interface stability is a hard constraint. It can't skip tests, because test passage is a gate in the validation step. The diff size limit forces incremental changes that are easy to review and revert.

The constraint file itself becomes documentation: "We preserve these interfaces, we require these test patterns, we don't modify more than N files per change." New engineers learn the architecture by reading the constraints. The assistant and the humans are working from the same rulebook.

Building your own constraint layer

If you're running RAG or fine-tuning today, the path forward isn't ripping everything out—it's adding validation gates. Before any generated code reaches a human, run it through:

1. **Static analysis at PR time**: ESLint, mypy, cargo clippy, whatever your language offers. If the generated code doesn't pass your normal CI checks, reject it automatically.

2. **Explicit interface contracts**: Maintain a list of stable interfaces/APIs that cannot be modified without approval. Flag any generated diff that touches these as high-risk.

3. **Mandatory test coverage**: If the change adds logic, it must add corresponding tests. No "we'll test it later."

The Goatfied agent loop bakes these in (plan → constrain → edit → validate → retry), but you can retrofit them onto any system by treating generated code as untrusted input until it proves itself. Self-hosted Goatfied setups often extend this with custom validators that encode team-specific rules: "no new database migrations in minor releases," "all API changes require OpenAPI spec updates," "crypto operations must use the approved library."

The constraint file lives in version control, changes go through code review, and everyone—human and AI—works against the same standards. When the standards change, you update the constraints, not retrain a model.

Related reading

• [Why we open-sourced Goatfied (and what we kept proprietary)](/blog/why-we-open-sourced-goatfied-and-what-we-kept-proprietary)
• [Open-Source playbook #3: practical Goatfied tactics for shipping PRs](/blog/goatfied-open-source-playbook-3)