Why small models win at completion and lose at refactoring

A 1B–3B parameter model running locally can autocomplete your function faster than you can type it. The same model will confidently rewrite your entire module in a way that compiles but breaks three downstream services. Understanding why reveals fundamental differences in how language models approach narrow versus broad tasks—and which problems are worth solving with smaller, faster inference.

The performance gap isn't about model quality in the abstract. It's about context windows, retrieval boundaries, and the difference between generating twenty tokens that follow established patterns versus reasoning about fifty files you haven't opened yet.

Completion favors shallow context and high throughput

Autocomplete—whether single-line or function-body—succeeds when the model can lean on immediate context. You're inside a function, the type signatures are visible, the variable names are in scope, and the model needs to produce a dozen tokens that satisfy local constraints.

Small models excel here because:

Local context is dense. The preceding ten lines contain almost everything needed: the function signature, the variables already declared, the return type. A 2K token window is enough to see the method you're implementing, the class it belongs to, and the imports at the top of the file.

Patterns are repetitive. Iterating over a collection, null-checking before dereferencing, mapping API responses to domain objects—these appear thousands of times in training data. A smaller model's limited capacity is fine when the task is "recognize this setup and emit the statistically likely continuation."

Latency matters more than perfection. A completion that appears in 40ms feels like magic. One that takes 800ms disrupts flow. Small models running on local GPUs or quantized on CPU hit the lower bound; larger models, even with speculative decoding, pay for their parameter count in milliseconds.

The concrete tradeoff: a 3B model might suggest users.map(u => u.email) when you actually wanted users.filter(u => u.active).map(u => u.email), but you can fix that in two keystrokes. The speed of the feedback loop compensates for minor inaccuracies.

Refactoring demands graph traversal and constraint juggling

Renaming a function isn't just find-and-replace. You need to:

• Locate every call site across multiple files, some of which import the function indirectly through barrel exports.
• Distinguish between shadowed local variables that happen to share the name and actual references to the target symbol.
• Update tests that mock the function, configuration files that reference it by string, and documentation examples.

This is a graph problem disguised as a text problem. The model needs to build a mental map of dependencies, reason about symbol resolution rules, and make coordinated edits that preserve compile-time and runtime invariants.

Small models lack the capacity to hold the graph. A 1B parameter model has enough "working memory" to track a few variables and their immediate relationships. Refactoring a moderately complex codebase means juggling dozens of files, hundreds of symbols, and transitive dependencies the model has never seen in its context window.

Multi-hop reasoning fails silently. The model might correctly identify that processOrder is called in OrderService, but miss that OrderService itself is instantiated in three different microservices via dependency injection. It renames the method in the definition and the direct callers, ships the diff, and breaks integration tests.

Context window limits force lossy summarization. Even if you feed the model a 4K token summary of "all files that might reference this function," you've already discarded the specifics it needs: the exact import paths, the conditional logic around each call site, the configuration overrides in test fixtures.

We've seen refactors from small models succeed when the scope is narrow—renaming a private method used only within its defining class—and fail catastrophically when the change crosses module boundaries.

The retrieval problem compounds at scale

Completion works because relevant context is right there in the viewport. Refactoring requires pulling in files you haven't opened, which means retrieval. You need to find every file that imports the target, every test that exercises it, every config that mentions it.

Classic retrieval-augmented generation (RAG) uses embeddings to fetch "relevant" chunks, but code isn't prose. Semantic similarity between a function definition and its call sites is often low in embedding space—they don't share vocabulary, they share dependency relationships. A test file that imports OrderService might not mention "order" or "payment" in the actual test body; it's only relevant because of the import graph.

Small models compounded the issue: even if your retrieval is perfect and you surface exactly the six files that matter, cramming them into context leaves little room for reasoning. The model sees fragments, not the whole picture, and generates edits that are locally plausible but globally broken.

Larger models (30B+) can hold more of the graph in context and recover from imperfect retrieval by reasoning about what's missing. Smaller models treat the retrieved context as ground truth and hallucinate connections that don't exist.

When to reach for small models anyway

None of this means small models are useless beyond autocomplete. There are refactoring-adjacent tasks where they shine:

Localized rewrites with explicit constraints. "Convert this function to async/await" works well if the function is self-contained and you provide the new signature. The model isn't traversing the call graph—you've scoped the problem to a single definition.

Formatting and style fixes. Rewriting a block of imperative code to use a pipeline of functional methods, extracting magic numbers to named constants, splitting a hundred-line function into smaller helpers—all of these are "refactoring" in the colloquial sense but don't require cross-file reasoning.

Generating test scaffolding. Given a function signature, a small model can produce a reasonable test skeleton: the imports, the setup, a handful of assertions. You'll fill in edge cases and mocks, but the boilerplate is handled.

The pattern: small models succeed when the task is generative and locally scoped, not when it's analytical and graph-traversal-heavy.

How Goatfied constrains the blast radius

Our agent loop (plan → constrain → edit → validate → retry) enforces compile and lint gates before any code is committed. That means even when a smaller model suggests a refactor that misses call sites, the validation step catches the undefined references and forces a retry with explicit feedback: "compilation failed in OrderService.ts, line 47: cannot find name 'processOrder'."

The constraint phase also narrows scope by asking "which files must change together for this to be a valid edit?" before generating diffs. If the model proposes renaming a public export, the planner surfaces all importers and adds them to the edit set, turning a multi-hop reasoning problem into a series of localized edits with visible dependencies.

This doesn't magically make a 3B model as capable as a 70B model, but it shifts the bottleneck. Instead of "can the model hold the entire call graph in context," the question becomes "can the model make valid local edits given explicit constraints and fast feedback." Small models are much better at the latter.

We also support self-hosted deployments where you can run larger models for refactoring and smaller models for completion, routing requests based on task type. Managed Goatfied does this automatically: we use a small, quantized model for sub-200ms completions and a larger reasoning model for multi-file edits where latency is less critical than correctness.

Picking the right model for the job

The heuristic: if the model needs to generate tokens that follow from visible context, small is fast and good enough. If the model needs to reason about invisible dependencies or make coordinated changes, you need more capacity.

Concretely:

• Autocomplete: 1–3B models, locally hosted or fast API.
• Function/method body generation: 3–7B models work; 13B+ helps with complex logic.
• Cross-file refactoring: 30B+ or route to a large API model (GPT-4, Claude Opus, etc.).
• Codebase-wide migrations: large models with long context windows, paired with static analysis tools to validate edits.

There's no free lunch. Small models buy you speed and local inference at the cost of reasoning depth. Treating them as drop-in replacements for large models on complex tasks wastes time on retry loops and broken diffs. Treating them as unsuitable for anything but autocomplete ignores genuinely useful applications where throughput and latency matter more than exhaustive correctness.

The win is knowing which problem you're solving and matching the model capacity to the task's actual requirements—not the marketing deck's promises.

Related reading

• When fine-tuning beats RAG for code, and when it doesn't
• Quantization tradeoffs for code models: what you actually lose