Completion Ranking With Repository Signals Failure Patterns And: practical systems guide

# Completion Ranking With Repository Signals Failure Patterns And: practical systems guide

When your AI assistant suggests five different ways to handle a database migration, how does it decide which completion to show first? Most developers assume the model just "knows" based on training data. In practice, production completion systems layer repository-specific signals on top of base model outputs—and the failure patterns that emerge tell you exactly where naive ranking falls apart.

This matters because bad rankings don't just waste time. They teach your agent loop to prefer patterns that compile but break at runtime, pass tests but violate team conventions, or work in isolation but conflict with recent changesets three directories over. Understanding how completion ranking actually works—and fails—helps you build agents that make fewer embarrassing mistakes.

Why base model perplexity isn't enough

Language models output a probability distribution over tokens. The simplest ranking approach sorts completions by cumulative log probability: whichever sequence the model assigns highest likelihood wins. This works reasonably well for isolated code snippets.

It breaks the moment you introduce repository context. A model might confidently suggest `fs.readFile()` in a Node.js codebase that standardized on `fs.promises.readFile()` six months ago. The raw model probability looks great—`fs.readFile` appears in millions of training examples. But the *local* signal (recent commits, lint rules, existing patterns) screams "wrong choice."

Repository-aware ranking layers additional signals on top of base probabilities:

• **Import frequency:** if 90% of files use `import { z } from 'zod'` and 10% use `import * as zod from 'zod'`, weight completions accordingly
• **Lint/type outcomes:** simulate running `tsc` or `eslint` on each candidate and penalize those that produce errors
• **Similarity to recent edits:** if the last five PRs all added error boundaries around async calls, prefer completions that follow that pattern
• **Test proximity:** rank higher completions that reference types/modules covered by existing tests

None of these guarantee correctness. They shift the distribution toward locally consistent choices.

Common failure pattern: stale import ranking

Imagine your team migrated from `lodash` to `lodash-es` for better tree-shaking. The codebase still contains 200 files importing from `lodash`, but all new code uses `lodash-es`. A frequency-based ranker that simply counts import statements across the repo will keep suggesting the deprecated pattern.

Better systems track *temporal signals*. Weight recent commits higher when computing import frequencies. In Goatfied's agent loop, the **constrain** step surfaces lint rules and type constraints before the edit happens, so completions that would introduce `lodash` imports get penalized at ranking time, not after the fact.

Practical fix: maintain a sliding window (e.g., last 90 days of commits) when building import frequency tables. Or better, hook into your lint config—if you've added `no-restricted-imports` rules, use those as hard constraints during ranking.

When type-checking signals mislead

Running `tsc --noEmit` on each completion candidate sounds great: rank by how many type errors each produces. In practice, this creates a new class of failures.

Type errors often cascade. A completion might introduce zero errors *locally* but break type inference three files away because it narrows a union type unexpectedly. The ranker sees a clean compile and ranks it high. Later, when the agent runs the full validation step, the build explodes.

Another edge case: overly permissive types. A completion that casts everything to `any` will produce zero type errors. A naive type-error-based ranker might prefer it over a completion that correctly uses a union type but triggers one inference error in a distant generic.

The fix isn't to skip type-checking signals—they catch too many real errors. Instead:

• Run type checks on the minimal changed scope *plus* immediate dependents, not just the edited file
• Penalize completions that introduce type assertions or `any` casts unless they're removing more type errors than they add
• Use type error *count* as a tiebreaker, not the primary signal

In the Goatfied loop, we run constrain-time type checks on the edit target, then validation-time checks on the full affected scope. This two-phase approach catches local errors fast while deferring whole-program analysis until after you've committed to a candidate.

Test coverage blind spots

Ranking systems often boost completions that reference functions/types covered by tests. The logic: if `processPayment()` has 20 test cases, a completion calling that function is probably safer than one calling the untested `legacyProcessPayment()`.

This heuristic fails when tests exist but don't exercise relevant edge cases. Your `processPayment()` tests might cover happy paths beautifully but ignore decimal precision bugs. A completion that rounds currency values incorrectly will rank high because it calls the "well-tested" function, even though the specific usage pattern has zero coverage.

Slightly better: rank by *relevant* test coverage. If the completion includes currency math, check whether tests exercise decimal precision specifically. This requires parsing test bodies to extract assertions, which is expensive but catches real bugs.

An intermediate approach: track failure patterns from CI. If `processPayment()` appears in 15 flaky test failures over the past month, penalize completions that increase call sites without adding error handling. This doesn't require deep test analysis—just correlating symbols to historical failures.

Balancing exploration vs. exploitation

Sophisticated rankers introduce *exploration*: occasionally ranking a lower-probability completion higher to gather signal. This prevents local maxima where your agent keeps suggesting the same five patterns because those have historically been safe.

Set a small randomization budget—maybe 5% of completions bypass normal ranking and sample from the broader distribution. Track outcomes: does the agent's edit pass validation? Does it get accepted by reviewers? Feed this back into future ranking.

Without exploration, teams get stuck. Your ranking system learns that `if (err) { return null; }` always compiles cleanly, so it dominates suggestions. Your team never discovers that `if (err) { throw new AppError(err.message); }` would be better because the ranker never surfaces it.

Goatfied's agent loop handles this by letting you configure retry strategies. If the top-ranked completion fails validation, the agent samples from lower-ranked candidates before giving up. Over time, this surfaces alternatives that naive ranking missed.

Practical implementation checklist

If you're building or tuning a completion ranker:

1. **Separate constraint enforcement from ranking**—hard failures (type errors, lint violations) should block completions entirely, not just downrank them

2. **Weight recent repository activity higher** when computing frequency-based signals

3. **Run type checks on minimal scopes** during ranking, full scopes during validation

4. **Track temporal patterns in test failures** to penalize brittle call sites

5. **Reserve 5–10% of suggestions for exploration** to avoid local maxima

6. **Log ranking decisions** so you can audit why the agent chose what it did

None of this is exotic ML research. It's plumbing: connecting the model's output to the specific, measurable properties of your codebase. The teams that ship reliable agents aren't using better models—they're using better signals.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)