Completion Ranking With Repository Signals Production Playbook: practical systems guide

When your AI editor suggests completions in the middle of a 200k-line codebase, ranking matters more than raw retrieval. You can feed every remotely relevant symbol into context, but the first three suggestions drive 80% of acceptance. Poor ranking turns a 300ms completion into a 15-second scroll-and-search session where developers lose flow and trust evaporates.

Repository signals—import frequency, call-graph centrality, recent edit patterns, test coverage, module size—let you distinguish between the utility function everyone depends on and the deprecated wrapper buried in `/legacy`. Integrating these signals into production completion ranking is straightforward in concept but deceptively hard in practice. This guide walks through the specific systems decisions we made at Goatfied to keep ranking fast, accurate, and auditable.

Why repository signals beat embeddings alone

Vector similarity excels at semantic matching but treats all code equally. A helper function used in three files and your core authentication module both get embedded with similar care. Repository signals provide structural weight: how often is this imported? How central is it in the call graph? Was it edited in the last sprint?

The combination lets you surface `getUserPermissions()` before `getUserPermissionsLegacyV2Deprecated()` even when both match the query embedding well. In the Goatfied agent loop—plan, constrain, edit, validate, retry—this matters during the *constrain* and *edit* phases. If the agent picks the wrong module on its first attempt, validation (compile + lint + test) catches the error, but the retry cycle costs time. Better ranking reduces retries.

We don't invent numbers, but directionally: ranking by embeddings alone caused noticeably more "imported the wrong variant" errors in early iterations. Adding import-frequency and recent-edit signals cut those errors enough to make the agent loop feel tighter.

Computing signals without crushing CI

Repository signals need updates on every commit, but recalculating everything on every push doesn't scale past a few hundred files. We split signals into three tiers by update cost:

**Hot signals** (updated every commit): recent edit timestamps, changed-file lists, and lightweight import counts extracted directly from the commit diff. These run in a post-commit hook and write to a Redis sorted set keyed by file path. Total overhead: ~50ms for typical commits.

**Warm signals** (updated every 10 commits or hourly): call-graph centrality and cross-module import frequencies. We use a background worker that diffs the call graph incrementally—only re-analyzing files touched since the last run—and updates centrality scores with a single-source shortest-path approximation rather than full PageRank. This keeps warm updates under a few seconds even in repos with thousands of modules.

**Cold signals** (updated nightly): module size histograms, test coverage maps, and dead-code detection. These batch jobs run in off-peak hours and write to Postgres. They inform ranking but don't need real-time freshness.

The tier system keeps the critical path fast. When a developer requests a completion, we read hot signals from Redis (sub-millisecond), warm signals from an in-memory cache refreshed every minute, and cold signals from a materialized Postgres view.

Ranking formula that survives edge cases

We use a weighted sum with manual coefficients tuned on internal dogfooding. The base formula looks like:

score = (

    0.4 * embedding_similarity

    + 0.25 * log(import_frequency + 1)

    + 0.15 * centrality_percentile

    + 0.1 * recency_weight

    + 0.1 * test_coverage_bonus

)

Logarithmic scaling on import frequency prevents a single heavily-used utility from dominating every query. Recency weight decays exponentially: files edited in the last day get a 2× boost, last week 1.5×, older than a month no boost. Test coverage adds a small bonus to nudge developers toward well-tested modules, but we cap it so untested-but-correct code isn't penalized out of the top ten.

Edge cases that broke early versions:

• **Circular imports**: Two modules importing each other spiked each other's scores. We now detect cycles during call-graph analysis and cap their combined weight.
• **Refactor storms**: A large rename touches 200 files, inflating recency for everything. We detect high-churn commits (files changed > 50 in one push) and discount recency by 50% for those events.
• **Monorepo vendors**: Third-party code in `/vendor` or `/node_modules` sometimes has high import frequency but low relevance. We exclude paths matching common vendor patterns from import-frequency ranking.

Coefficients aren't sacred. We log every completion acceptance (which suggestion was picked, at what rank) and re-tune quarterly. The current weights reflect our Go and TypeScript codebases; Python repos might need higher centrality weight due to less explicit import discipline.

Keeping ranking decisions auditable

When a completion is wrong, developers need to understand *why* it ranked first. We store the full signal breakdown for each suggestion in the completion response payload:

{

  "symbol": "pkg.Authenticate",

  "rank": 1,

  "embedding_score": 0.92,

  "import_freq": 47,

  "centrality": 0.85,

  "last_edit": "2025-01-10T14:22:00Z",

  "test_coverage": 0.78

}

This shows up in Goatfied's completion UI as an expandable detail panel. If `Authenticate` ranks above `AuthenticateV2` unexpectedly, an engineer can see that centrality is driving the difference and file an issue if that's wrong.

Auditability also helps with compliance workflows. Some teams require all AI-suggested code to include a provenance trail. Surfacing which signals influenced ranking—especially for self-hosted Goatfied deployments—makes that trail traceable without reverse-engineering the model.

Incremental rollout and fallback

We deployed ranking signals in three phases:

1. **Shadow mode**: Compute signals and log proposed rankings alongside embedding-only rankings, but serve only embeddings. Compare logs to measure how often signals would change the top result.

2. **Partial rollout**: Serve signal-informed ranking to 10% of completions, flagged with a `signals_enabled=true` tag. Monitor acceptance rates and retry counts in the agent loop.

3. **Full rollout**: Enable for all completions, with a feature flag to fall back to embeddings-only if signal computation times out (>100ms).

The fallback is critical for reliability. If Redis is unavailable or the call-graph worker is behind, we degrade gracefully to embeddings rather than blocking completions entirely. This aligns with Goatfied's compile-first philosophy: tools should fail safe, not fail silent or fail broken.

Measuring ranking improvements

We track two primary metrics:

• **Top-3 acceptance rate**: Percentage of accepted completions that appeared in the first three suggestions. This rose from ~68% with embeddings alone to ~79% after adding signals (measured over 4 weeks, internal dogfooding, ~15k completions).
• **Agent-loop retries per PR**: Average validate-and-retry cycles before a clean compile. Signal-informed ranking didn't eliminate retries—type errors and logic bugs still happen—but reduced "imported wrong variant" retries by roughly half.

Both metrics come from internal usage, not customer benchmarks. We call them out as dogfooding observations, not universal guarantees. Your repo structure, language mix, and team habits will shift results.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)