Completion Ranking With Repository Signals Implementation Guide: practical systems guide

When your autocomplete engine serves ten possible continuations, choosing which one to show first isn't a ranking problem—it's a context problem. The model thinks `user.getName()` and `user.getEmail()` are equally plausible, but your repository knows that `getName()` appears in exactly two files while `getEmail()` gets called 847 times across 23 modules. That signal alone can flip a useless suggestion into the right answer.

Traditional completion rankers lean on language model confidence scores, maybe some recency heuristics. But the repository itself—call graphs, import patterns, test coverage, recent edit history—holds structural truth that pure statistical models miss. The challenge is surfacing those signals without grinding autocomplete latency into the ground or shipping a thousand-line feature flag maze.

Collecting signals cheaply at index time

Fast ranking starts with cheap lookups. Build a lightweight signal index alongside your existing code intelligence infrastructure rather than computing everything on-demand. For a typical repository, track:

• **Symbol reference counts**: how many times each function, class, or variable appears across all files. Store these in a simple key-value map: `"AuthService.validateToken" -> 412`.
• **Co-occurrence within files**: which symbols frequently appear together in the same file. If `logger.info` and `metrics.track` show up in 80% of the same files, that's a co-location signal.
• **Recency weights**: timestamp the last edit to each file, then propagate a decay score to every symbol defined or called in that file. A function touched yesterday scores higher than one untouched for six months.

Build this index incrementally on file changes rather than full-repo scans. When a developer edits `api/auth.ts`, recompute signals only for symbols defined or referenced in that file. Store the index in something fast—SQLite, even just a JSON blob if your repo is under 100k lines. The goal is single-digit millisecond lookup when the autocomplete request arrives.

// Minimal signal store interface

interface SignalStore {

  getReferenceCount(symbol: string): number;

  getCoOccurrence(symbolA: string, symbolB: string): number;

  getRecencyScore(symbol: string): number;

}

Ranking formula: additive scoring beats black-box tuning

Don't train a neural ranker unless you have millions of labeled examples and weeks to burn. Start with a simple additive score that combines three components:

rank_score = (model_confidence * 0.5)

           + (log(ref_count + 1) * 0.3)

           + (recency_weight * 0.2)

The model confidence comes from your language model's raw logits. Reference count gets log-scaled so the difference between 10 and 100 references matters more than 1000 vs 1100. Recency weight decays exponentially: `exp(-days_since_edit / 30)` keeps recently touched code relevant.

These coefficients are starting points. Tune them by sampling 50 real autocomplete sessions, manually labeling which completion you would've picked, then hill-climbing the weights to maximize your chosen metric. This takes an afternoon, not a quarter.

The beauty of additive scoring: you can explain why a completion ranked first. "This method has 380 references, was edited 3 days ago, and the model gave it 72% confidence." Ship that explanation in a debug panel so developers trust the system.

Contextual signals: cursor position and import scope

The generic ranking formula gets you 80% of the way. The last 20% comes from request-time context:

• **Current file's import list**: if the file already imports `lodash/debounce`, bump `debounce` completions by 0.3. The developer has explicitly declared this dependency.
• **Cursor depth**: inside a conditional block three levels deep, prioritize recently used variables in that scope over globals. Parse the AST up to the cursor to extract local symbol availability.
• **Neighboring edits**: if the developer just typed `if (user.isAdmin())` two lines up, boost completions related to `user` properties.

These are boolean or small integer features. Multiply them into your base score or add them as fixed deltas. Resist the urge to overthink—a 0.2 boost for "symbol in current imports" is plenty.

Latency: the 50ms budget

Autocomplete ranking must finish in under 50ms or developers will notice the lag. Profile every step:

• Model inference: 20–30ms (this is mostly fixed, optimize by batching requests or serving from local model cache)
• Signal lookups: 5–10ms (SQLite indexed queries, or in-memory map reads)
• Ranking arithmetic: <1ms (pure CPU math)

If signal lookups creep past 10ms, you're doing something wrong. Pre-load the signal index into memory on editor startup. For large monorepos, shard the index by directory and only load shards relevant to the currently open file.

On Goatfied, we fold these signal lookups into the same compilation pass that validates syntax. The agent loop already parses the file to check if edits compile—extracting symbol references and recency scores adds negligible overhead. This way, ranking signals stay fresh without extra I/O.

Validation: measure accept rate, not model perplexity

Track what developers actually accept, not what the model thought was probable. Instrument your completion UI to log:

• **Accept rate by rank**: what percentage of shown completions at rank 1, 2, 3 get accepted?
• **Time to accept**: if developers hesitate for 5 seconds before accepting rank 1, maybe rank 2 was actually better.
• **Edit-after-accept**: if 40% of accepted completions get immediately edited, your ranking is serving close-but-wrong suggestions.

Set a baseline before adding repository signals, then measure the lift. A good signal set moves 10–15% of accepts from rank 2–3 up to rank 1. If you see no change, either your signals are noisy or your baseline ranker was already strong.

Avoid vanity metrics like "model accuracy" on held-out data. Developers don't care if your ranker matches a labeled dataset—they care if the first suggestion is the one they want.

Iteration loop: weekly signal experiments

Once the core system is running, iterate on new signals:

• **Git blame recency**: prioritize symbols edited by the current developer in the past week.
• **Test coverage**: boost symbols with existing tests, since developers are likelier to reuse well-tested code.
• **Package version alignment**: if the repo uses `express@4.18`, rank completions using Express 4.x APIs higher than deprecated 3.x patterns.

Add one signal per week, measure accept rate lift, and keep it if the gain exceeds 2%. Ship small, reversible changes—Goatfied's self-hosted mode lets you roll back a ranking tweak in one command if it regresses developer experience.

The goal isn't a perfect ranker. It's a system that improves every week and makes autocomplete feel like it reads your mind, because it's reading your repository.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)