Completion ranking with repository signals

When your AI assistant suggests ten ways to handle a database query, choosing which appears first isn't a language model problem—it's a repository problem. The model might assign 40% confidence to `db.query()` and 38% to `db.rawQuery()`, but your codebase knows that `rawQuery()` was deprecated three months ago and appears in exactly two legacy files while `query()` is called 847 times across active modules. That structural signal turns a coin flip into the right answer.

Most completion systems rank by raw model probability, maybe layering in recency heuristics. But the repository itself—call graphs, import patterns, test coverage, recent commit history—holds ground truth that pure statistical models miss. The challenge is surfacing those signals without grinding autocomplete latency into the ground or building a thousand-line feature flag maze.

The core ranking problem: probability vs. validity

Code completion systems generate multiple candidates through beam search or sampling. Each gets a probability reflecting "how likely is this token sequence given my training data," not "does this compile in *this* repository."

Consider a Python function needing a type annotation. The model suggests `-> List[str]` with 40% confidence and `-> list[str]` with 35%. If your `pyproject.toml` sets `target-version = "py39"` and you've imported `List` from `typing` everywhere, the capital-L version is the only valid choice. The probability difference is noise; the repository signal is the real discriminator.

This matters because bad rankings don't just waste time—they teach your agent to prefer patterns that compile but break at runtime, pass tests but violate team conventions, or work in isolation but conflict with changesets three directories over. Production completion systems layer repository-specific signals on top of base model outputs:

• **Import frequency:** If 90% of files use `import { z } from 'zod'` and 10% use `import * as zod`, weight completions accordingly
• **Type compatibility:** Does the completion match expected types at this position?
• **Lint/style conformance:** Does it follow project linting rules and naming conventions?
• **Call-graph centrality:** How often is this symbol referenced across the codebase?
• **Temporal signals:** Weight recent edits higher when computing pattern frequencies

Each heuristic has a cost. Parsing is cheap; running a type checker per candidate is expensive. The design question is which signals to compute and in what order.

Failure pattern: stale import ranking

Imagine your team migrated from `lodash` to `lodash-es` for better tree-shaking. The codebase still contains 200 files importing from `lodash`, but all new code uses `lodash-es`. A frequency-based ranker that simply counts import statements across the repo will keep suggesting the deprecated pattern.

Better systems track temporal signals. Weight recent commits higher when computing import frequencies. Maintain a sliding window—say, last 90 days of commits—when building import frequency tables. Or hook directly into your lint config: if you've added `no-restricted-imports` rules, use those as hard constraints during ranking, not post-facto penalties.

In Goatfied's agent loop—plan → constrain → edit → validate → retry—the **constrain** step surfaces lint rules and type constraints before the edit happens, so completions that would introduce deprecated imports get penalized at ranking time.

Tradeoff one: latency vs. depth

The gold standard would be: generate candidates, compile each in full context, run tests, rank by "most likely to merge." This is prohibitive for interactive completion. Production systems front-load cheap checks instead.

A reasonable pipeline:

1. Filter syntactically invalid candidates immediately (sub-millisecond)

2. Run lightweight type checks on the remaining five to ten candidates (10–50ms)

3. Surface the top result; speculatively validate runners-up in the background

Completion ranking must finish in under 50–100ms or developers notice lag. Profile every step:

• Model inference: 20–30ms (mostly fixed; optimize by batching or serving from local cache)
• Signal lookups: 5–10ms (SQLite indexed queries or in-memory map reads)
• Ranking arithmetic: <1ms (pure CPU math)

If signal lookups exceed 10ms, you're doing something wrong. Pre-load the signal index into memory on editor startup. For large monorepos, shard the index by directory and only load shards relevant to the currently open file.

At Goatfied, we fold signal lookups into the same compilation pass that validates syntax. The agent loop already parses the file to check if edits compile—extracting symbol references and recency scores adds negligible overhead.

The tradeoff: lower latency ranking might promote a subtly broken completion; the validation gate catches it, but you've spent a retry cycle. Higher latency ranking (e.g., running local type checks per candidate before showing anything) delays feedback but reduces retry noise. For repositories with fast compile times—Go modules, Rust crates with incremental builds—running a real compile per candidate becomes feasible. For slow monorepos, you need proxies: cached type information, partial builds, or pattern matching against known-good diffs.

Failure pattern: type-checking signals that mislead

Running `tsc --noEmit` on each completion candidate sounds great: rank by fewest type errors. In practice, this creates new failure modes.

Type errors cascade. A completion might introduce zero errors locally but break type inference three files away because it narrows a union type unexpectedly. The ranker sees a clean compile and ranks it high. Later, when the agent runs full validation, the build explodes.

Another edge case: overly permissive types. A completion casting everything to `any` produces zero type errors. A naive error-based ranker might prefer it over a completion that correctly uses a union type but triggers one inference error in a distant generic.

The fix isn't to skip type-checking signals—they catch too many real errors. Instead:

• Run type checks on the minimal changed scope *plus* immediate dependents, not just the edited file
• Penalize completions introducing type assertions or `any` casts unless they remove more errors than they add
• Use type error count as a tiebreaker, not the primary signal

In the Goatfied loop, we run constrain-time type checks on the edit target, then validation-time checks on the full affected scope. This two-phase approach catches local errors fast while deferring whole-program analysis until after committing to a candidate.

Building the signal index: what to track and how

Fast ranking starts with cheap lookups. Build a lightweight signal index alongside your existing code intelligence infrastructure rather than computing everything on-demand. For a typical repository, track:

• **Symbol reference counts:** How many times each function, class, or variable appears across files. Store in a simple key-value map: `"AuthService.validateToken" -> 412`
• **Co-occurrence within files:** Which symbols frequently appear together. If `logger.info` and `metrics.track` show up in 80% of the same files, that's a co-location signal
• **Recency weights:** Timestamp the last edit to each file, then propagate a decay score to every symbol defined or called there. A function touched yesterday scores higher than one untouched for six months

Build this index incrementally on file changes rather than full-repo scans. When a developer edits `api/auth.ts`, re-index that file and immediate dependents—not the world. Store the index in something fast—SQLite, even a JSON blob for repos under 100k lines. The goal is single-digit millisecond lookup when the autocomplete request arrives.

interface SignalStore {

  getReferenceCount(symbol: string): number;

  getCoOccurrence(symbolA: string, symbolB: string): number;

  getRecencyScore(symbol: string): number;

}

Split signals into tiers by update cost:

**Hot signals** (every commit): Recent edit timestamps, changed-file lists, lightweight import counts extracted from the commit diff. Run in a post-commit hook, write to Redis. Overhead: ~50ms for typical commits.

**Warm signals** (every 10 commits or hourly): Call-graph centrality, cross-module import frequencies. Use a background worker that diffs the call graph incrementally—only re-analyzing touched files. Keeps updates under a few seconds even in repos with thousands of modules.

**Cold signals** (nightly): Module size histograms, test coverage maps, dead-code detection. Batch jobs run off-peak, write to Postgres. Inform ranking but don't need real-time freshness.

Practical ranking formula: additive scoring beats black-box tuning

Don't train a neural ranker unless you have millions of labeled examples and weeks to burn. Start with a simple additive score combining three to five components:

score = (

    0.4 * embedding_similarity

    + 0.25 * log(import_frequency + 1)

    + 0.15 * centrality_percentile

    + 0.1 * recency_weight

    + 0.1 * test_coverage_bonus

)

Logarithmic scaling on import frequency prevents a single heavily-used utility from dominating every query. Recency weight decays exponentially: files edited in the last day get a 2× boost, last week 1.5×, older than a month no boost. Test coverage adds a small bonus to nudge developers toward well-tested modules, but cap it so untested-but-correct code isn't penalized out of the top ten.

The beauty of additive scoring: you can explain why a completion ranked first. "This method has 380 references, was edited 3 days ago, and the model gave it 72% confidence." Ship that explanation in a debug panel so developers trust the system.

These coefficients are starting points. Tune by sampling 50 real autocomplete sessions, manually labeling which completion you would've picked, then hill-climbing the weights to maximize your chosen metric. This takes an afternoon, not a quarter.

Contextual signals: cursor position and import scope

Generic ranking formulas get you 80% of the way. The last 20% comes from request-time context:

• **Current file's import list:** If the file already imports `lodash/debounce`, bump `debounce` completions by 0.3. The developer has explicitly declared this dependency
• **Cursor depth:** Inside a conditional block three levels deep, prioritize recently used variables in that scope over globals. Parse the AST up to the cursor to extract local symbol availability
• **Neighboring edits:** If the developer just typed `if (user.isAdmin())` two lines up, boost completions related to `user` properties

These are boolean or small integer features. Multiply them into your base score or add them as fixed deltas. A 0.2 boost for "symbol in current imports" is plenty.

Edge cases that break naive rankers

**Circular imports:** Two modules importing each other spike each other's scores. Detect cycles during call-graph analysis and cap their combined weight.

**Refactor storms:** A large rename touches 200 files, inflating recency for everything. Detect high-churn commits (files changed > 50 in one push) and discount recency by 50% for those events.

**Large generated files:** GraphQL schemas, protobuf bindings pollute frequency counts. Filter files over 5,000 lines from usage stats unless explicitly imported.

**Vendored dependencies:** `node_modules/`, `vendor/`, `.venv/` should never contribute to repository-wide usage counts. Index them for type information but exclude from frequency and recency signals.

**Test file skew:** A flaky test edited hourly shouldn't make test helpers top suggestions in production code. Tag files matching `*.test.*`, `*_test.*`, `test_*` patterns and halve their recency weight.

Tradeoff two: local signals vs. global repository structure

Local signals are cheap: the function signature above the cursor, the imports at the top of the file, the variable names in scope. Global signals are expensive but powerful: the module's test coverage, the presence of a `Makefile` target enforcing schema validation, commit history showing which patterns have been refactored away.

A practical middle ground:

• **Indexed global signals:** At repository indexing time (on clone, on main-branch push), extract patterns like "all DAO classes inherit from `BaseModel`" or "API handlers always return `Result<T, ApiError>`." Store as metadata.
• **Lazy global lookups:** If a candidate references a symbol not in the current file, check the repository index for its definition and recent usage. This can be sub-50ms if indexed properly.

Goatfied's compile-first architecture makes this easier: because every edit is validated before commit, you have a continuous stream of "this diff compiled cleanly" vs. "this diff failed at lint" signals. Over time, these signals train a lightweight ranker: completions resembling successful diffs rank higher than those resembling rejected diffs.

The tradeoff: global signals require upfront indexing work and storage. For a 100k-line repository, a well-tuned index might be a few megabytes. For a 10M-line monorepo, you're managing gigabytes and need incremental update strategies.

Measuring and iterating on ranking

Track what developers actually accept, not what the model thought was probable. Instrument your completion UI to log:

• **Accept rate by rank:** What percentage of shown completions at rank 1, 2, 3 get accepted?
• **Time to accept:** If developers hesitate for 5 seconds before accepting rank 1, maybe rank 2 was actually better
• **Edit-after-accept:** If 40% of accepted completions get immediately edited, your ranking is serving close-but-wrong suggestions

Set a baseline before adding repository signals, then measure the lift. A good signal set moves 10–15% of accepts from rank 2–3 up to rank 1. If you see no change, either your signals are noisy or your baseline ranker was already strong.

Avoid vanity metrics like "model accuracy" on held-out data. Developers don't care if your ranker matches a labeled dataset—they care if the first suggestion is the one they want.

Once the core system is running, add one signal per week, measure accept rate lift, and keep it if the gain exceeds 2%. Ship small, reversible changes—Goatfied's self-hosted mode lets you roll back a ranking tweak in one command if it regresses developer experience.

Balancing exploration vs. exploitation

Sophisticated rankers introduce exploration: occasionally ranking a lower-probability completion higher to gather signal. This prevents local maxima where your agent keeps suggesting the same five patterns because those have historically been safe.

Set a small randomization budget—maybe 5% of completions bypass normal ranking and sample from the broader distribution. Track outcomes: does the agent's edit pass validation? Does it get accepted by reviewers? Feed this back into future ranking.

Without exploration, teams get stuck. Your ranking system learns that `if (err) { return null; }` always compiles cleanly, so it dominates suggestions. Your team never discovers that `if (err) { throw new AppError(err.message); }` would be better because the ranker never surfaces it.

Goatfied's agent loop handles this by letting you configure retry strategies. If the top-ranked completion fails validation, the agent samples from lower-ranked candidates before giving up. Over time, this surfaces alternatives that naive ranking missed.

Keeping ranking decisions auditable

When a completion is wrong, developers need to understand *why* it ranked first. Store the full signal breakdown for each suggestion in the completion response payload:

{

  "symbol": "pkg.Authenticate",

  "rank": 1,

  "embedding_score": 0.92,

  "import_freq": 47,

  "centrality": 0.85,

  "last_edit": "2025-01-10T14:22:00Z",

  "test_coverage": 0.78

}

This shows up in Goatfied's completion UI as an expandable detail panel. If `Authenticate` ranks above `AuthenticateV2` unexpectedly, an engineer can see that centrality is driving the difference and file an issue if that's wrong.

Auditability also helps with compliance workflows. Some teams require all AI-suggested code to include a provenance trail. Surfacing which signals influenced ranking—especially for self-hosted Goatfied deployments—makes that trail traceable without reverse-engineering the model.

Incremental rollout and graceful fallback

Deploy ranking signals in phases:

1. **Shadow mode:** Compute signals and log proposed rankings alongside embedding-only rankings, but serve only embeddings. Compare logs to measure how often signals would change the top result.

2. **Partial rollout:** Serve signal-informed ranking to 10% of completions, flagged with a `signals_enabled=true` tag. Monitor acceptance rates and retry counts in the agent loop.

3. **Full rollout:** Enable for all completions, with a feature flag to fall back to embeddings-only if signal computation times out (>100ms).

The fallback is critical for reliability. If Redis is unavailable or the call-graph worker is behind, degrade gracefully to embeddings rather than blocking completions entirely. This aligns with Goatfied's compile-first philosophy: tools should fail safe, not fail silent or fail broken.

Practical implementation checklist

If you're building or tuning a completion ranker:

1. **Separate constraint enforcement from ranking:** Hard failures (type errors, lint violations) should block completions entirely, not just downrank them

2. **Weight recent repository activity higher** when computing frequency-based signals

3. **Run type checks on minimal scopes** during ranking, full scopes during validation

4. **Track temporal patterns in test failures** to penalize brittle call sites

5. **Reserve 5–10% of suggestions for exploration** to avoid local maxima

6. **Log ranking decisions** so you can audit why the agent chose what it did

7. **Profile latency at every step:** If signal lookups creep past 10ms, you're doing I/O wrong

None of this is exotic ML research. It's plumbing: connecting the model's output to the specific, measurable properties of your codebase. The teams that ship reliable agents aren't using better models—they're using better signals.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)
• [Cursor, Copilot, Continue, Cody, Windsurf, Codeium: comparison framework](/blog/cursor-copilot-continue-cody-windsurf-codeium-comparison-framework)