Completion Ranking With Repository Signals Operational Checklist: practical systems guide

# Completion Ranking With Repository Signals Operational Checklist: practical systems guide

When your AI autocomplete suggests `getUserById()` instead of the deprecated `fetchUser()` three files away, that's repository-aware ranking at work. Most completion systems serve suggestions in a vacuum—lexical distance, recency, frequency. Adding repository context—import graphs, usage patterns, deprecation markers—turns completions from plausible to actually correct for your codebase.

This guide walks through building and operating a completion ranker that ingests repository signals. We'll cover indexing strategies, ranking heuristics that matter, and the operational checkpoints that keep the system useful as your codebase evolves.

Index what matters, skip the rest

Repository signals live in three buckets: structural (imports, call graphs, type definitions), temporal (commit frequency, recent edits), and semantic (comments, deprecation annotations, test coverage). Your indexer needs to extract these efficiently without rehashing the entire tree on every save.

Start with incremental parsing. When a file changes, re-index that file and immediate dependents—not the world. Use a language server protocol adapter if your stack supports it; LSP already maintains symbol tables and dependency graphs. Store symbols in a columnar format keyed by file hash so you can diff cheaply:

interface SymbolIndex {

  fileHash: string;

  symbols: Array<{

    name: string;

    kind: 'function' | 'class' | 'variable';

    exported: boolean;

    deprecated: boolean;

    importedFrom?: string[];

    usageCount: number;

  }>;

}

Track deprecation explicitly. Scan for `@deprecated` JSDoc tags, Python `DeprecationWarning` decorators, Rust `#[deprecated]` attributes. Flag these symbols with weight penalties at rank time. A deprecated function buried in legacy code shouldn't surface above the current API.

Usage frequency is gold. Count how often a symbol appears across the repo—not just definitions, actual call sites. A helper used in fifty files probably matters more than a one-off utility. Decay counts over time: symbols unused for three months drop in rank even if historically popular.

Ranking heuristics that survive contact with real code

Naive ranking concatenates scores: `lexical_match * 0.4 + recency * 0.3 + frequency * 0.3`. This breaks the moment you have a common variable name (`data`, `result`) that drowns out context-specific symbols.

Use a tiered filter instead. First pass: only symbols reachable from current file's import graph or same module. Second pass: boost symbols used in recently edited files (say, last 72 hours). Third pass: apply deprecation penalties and frequency weights. This keeps suggestions grounded in what the developer can actually use without hunting down imports.

Context window matters more than global popularity. If the user just typed `import { use` in a React file, rank hooks higher than unrelated `use*` functions from a database layer. Scan the current file's existing imports and weight symbols from those packages 2-3x higher than cold suggestions.

Handle monorepo boundaries carefully. A symbol popular in `services/payments` shouldn't auto-suggest in `frontend/dashboard` unless there's explicit cross-boundary usage. Track package or directory boundaries in your index and penalize cross-boundary suggestions unless the developer has imported from that boundary before.

Operational checkpoints

**Latency budget: sub-100ms p95.** Completion ranking runs in the hot path. If your index lookup + score computation exceeds 100ms, developers will type ahead and ignore suggestions. Use in-memory caches for frequently accessed symbol tables. Precompute usage counts nightly and store deltas for intraday updates.

**Stale index detection.** A common failure mode: the index falls behind, suggesting old APIs or missing new symbols. Instrument indexing lag—time between file save and index update—and alert when lag exceeds 10 seconds. If your indexer crashes or falls behind, completions silently degrade to lexical-only, and nobody notices until frustration accumulates.

**Deprecation audit trail.** When you mark a symbol deprecated, completions should stop surfacing it within one indexing cycle. Log every deprecated suggestion served; if developers keep seeing deprecated symbols, either the annotation isn't parsing or the ranking weights are wrong. Review these logs weekly.

**Cross-file consistency.** If `payments/client.ts` exports `processPayment()` but `payments/index.ts` re-exports it, both should rank identically. Normalize export chains in your index so the developer gets one clear suggestion, not duplicates from different file paths.

**A/B test ranking changes carefully.** Ranking adjustments feel minor—boost recency by 10%, penalize deprecated symbols harder—but they shift developer muscle memory. Roll out ranking changes to 10% of traffic first. Track acceptance rate (suggestion shown → accepted) before and after. A 5% drop in acceptance means the new ranking is worse, even if it "should" be better on paper.

Handling edge cases without special-casing everything

Large generated files (GraphQL schemas, protobuf bindings) pollute frequency counts. Filter files over 5,000 lines from usage stats unless explicitly imported. If a developer imports from a generated file, honor it, but don't let auto-generated symbols dominate global rankings.

Vendored dependencies create noise. `node_modules/`, `vendor/`, `.venv/` should never contribute to repository-wide usage counts. Index them for type information but exclude from frequency and recency signals.

Test files skew recency. A flaky test edited hourly shouldn't make test helpers top suggestions in production code. Tag files matching `*.test.*`, `*_test.*`, `test_*` patterns and halve their recency weight. Test utilities still surface when relevant, but don't overwhelm.

Integrating with the agent loop

In Goatfied's agent loop, completion ranking feeds the **constrain** phase. When the agent generates code, it uses repository signals to pick the right APIs on first try—no hallucinated function names, no deprecated imports. The validate phase then confirms the completions compile and pass lint.

This creates a feedback loop: completions that pass validation boost their ranking; hallucinated symbols that fail validation get logged and excluded from future suggestions. Over time, the ranker learns your repository's ground truth—what actually compiles—not just what looks plausible.

Track validation pass rates per symbol. If `processPayment()` from `payments/client.ts` validates 95% of the time but `processPayment()` from `legacy/payments.ts` fails validation 60% of the time, weight the former higher. This self-correcting signal beats hand-tuning ranking weights.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)