Completion Ranking With Repository Signals Design Tradeoffs: practical systems guide

When your AI-powered code completion model suggests ten plausible lines to complete a function, how do you pick which one surfaces first? The naive answer—"show the highest-probability token sequence"—falls apart the moment you look at real repository context: type signatures, import patterns, naming conventions, and the architectural constraints that make one completion correct and nine others subtly wrong.

Completion ranking turns out to be less about raw language model confidence and more about filtering suggestions through the signals already present in your codebase. Done well, ranking respects the compile/lint/test contract your team already enforces. Done poorly, it trains developers to accept plausible-looking code that breaks minutes later.

The core ranking problem: probability vs. validity

A code completion system generates candidates—typically beam search or sampling from a language model produces multiple options. The model assigns each a probability, but that probability reflects "how likely is this token sequence given my training data," not "does this pass CI in *this* repository."

Consider a Python function that needs a type annotation. The model might suggest `-> List[str]` with 40% confidence and `-> list[str]` with 35% confidence. If your repository's `pyproject.toml` sets `target-version = "py39"` and you've imported `List` from `typing` everywhere, the capital-L version is the only valid choice. The probability difference is noise; the repository signal is the real discriminator.

Ranking systems solve this by layering heuristics:

1. **Syntax validity:** Does the completion parse? If you're mid-expression, does it close brackets correctly?

2. **Type compatibility:** Does it match the expected type at this position?

3. **Style conformance:** Does it follow the project's linting rules and naming conventions?

4. **Import availability:** Are the symbols it references already imported or trivially importable?

5. **Semantic coherence:** Does it align with similar patterns elsewhere in the repo?

Each heuristic has a cost. Parsing is cheap; running a type checker per candidate is expensive. The design question is which signals to compute and in what order.

Tradeoff one: latency vs. depth

The gold standard would be: generate candidates, compile each in full context, run the test suite, rank by "most likely to merge." This is obviously prohibitive for interactive completion. Instead, systems front-load cheap checks.

A reasonable pipeline:

1. Filter out syntactically invalid candidates immediately (sub-millisecond).

2. Run lightweight type checks on the remaining five to ten candidates (10–50 ms).

3. Surface the top result; speculatively validate runners-up in the background.

At Goatfied, we lean on the **agent loop's validation gate**: completions don't just suggest code, they produce a diff that immediately hits `compile -> lint -> test`. If a completion fails any gate, the agent retries with the error message as additional context. This means we can afford to be slightly more aggressive in candidate generation because the loop itself penalizes bad suggestions harshly.

The tradeoff is: lower latency ranking might promote a subtly broken completion; the validation gate catches it, but you've spent a retry cycle. Higher latency ranking (e.g., running a local type check per candidate before showing anything) delays feedback but reduces retry noise.

For repositories with fast compile times—Go modules, Rust crates with incremental builds—running a real compile per candidate becomes feasible. For slow monorepos, you need proxies: cached type information, partial builds, or pattern matching against known-good diffs.

Tradeoff two: local signals vs. global repository structure

Local signals are cheap: the function signature above the cursor, the imports at the top of the file, the variable names in scope. Global signals are expensive but powerful: the module's test coverage, the presence of a `Makefile` target that enforces schema validation, the commit history showing which patterns have been refactored away.

A practical middle ground:

• **Indexed global signals:** At repository indexing time (on clone, on main-branch push), extract patterns like "all DAO classes inherit from `BaseModel`" or "API handlers always return `Result<T, ApiError>`." Store these as metadata.
• **Lazy global lookups:** If a candidate references a symbol not in the current file, check the repository index for its definition and recent usage. This can be a sub-50ms query if indexed properly.

Goatfied's compile-first architecture makes this easier: because every edit is validated before commit, you have a continuous stream of "this diff compiled cleanly" vs. "this diff failed at lint" signals. Over time, these signals train a lightweight ranker: completions that resemble successful diffs rank higher than those that resemble rejected diffs.

The tradeoff: global signals require upfront indexing work and storage. For a 100k-line repository, a well-tuned index might be a few megabytes. For a 10M-line monorepo, you're managing gigabytes and need incremental update strategies.

Tradeoff three: model-based vs. rule-based ranking

You can rank with explicit rules ("if the completion imports a deprecated symbol, penalize by 50 points") or train a learned ranker on historical acceptance data ("if completions with pattern X were accepted 80% of the time, boost X").

Learned rankers adapt to repository-specific style without manual tuning. The downside: they need labeled data (accepted vs. rejected completions) and can overfit to noisy signals (a developer accepting a completion because they were in a hurry, not because it was correct).

Rule-based rankers are transparent and debuggable. When a completion ranks poorly, you can trace exactly why. The downside: they require ongoing maintenance as languages and frameworks evolve.

A hybrid works well: use learned rankers for soft preferences (naming style, idiom choice) and hard rules for correctness (type mismatches, missing imports). At validation time, reject anything that fails hard rules regardless of model confidence.

In the Goatfied loop, the retry mechanism naturally generates training data: when a completion fails a gate and the agent fixes it, that failed → fixed pair is a clear signal for ranking. We don't need explicit accept/reject labels; the validation step provides them implicitly.

Practical implementation: start with compilation

If you're building or tuning a completion ranking system, the highest-ROI first step is: **make sure the top-ranked candidate compiles**. Not "looks plausible," but actually passes your build.

For statically typed languages, this is a type check. For dynamic languages, it's "imports resolve and linter passes." For configuration files, it's "schema validates."

Once you have that baseline, layer in softer signals: code style, idiom preference, commit history alignment. But never promote a completion that won't compile over one that will, regardless of model probability.

This principle is why Goatfied's agent loop insists on validation gates at every edit. It's easier to rank completions when you have an objective oracle—the compiler—telling you which suggestions are viable.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)