Repository Indexing That Keeps Context Fresh Implementation Guide: practical systems guide

Every AI coding assistant eventually hits the same wall: the model gives perfectly confident answers about code that was refactored three commits ago. You ask it to update a configuration file, and it hallucinates fields that existed in last week's schema. The root cause isn't the LLM—it's that your context is stale before the conversation even starts.

Repository indexing solves this by continuously parsing your codebase into a queryable representation the model can reference. But "indexing" is not a solved problem you install once. It's an ongoing systems challenge: what do you index, how often do you refresh it, and how do you keep the index cheap enough to run on every commit without turning your CI into a bottleneck?

This guide walks through building a repository indexing system that stays fresh without killing your build budget or latency. We'll cover the specific tradeoffs, the primitives you actually need, and the operational patterns that make this work in production.

Why staleness breaks agent reliability more than retrieval quality

Most indexing discussions focus on *what* you retrieve—embeddings, AST nodes, dependency graphs. But the failure mode that kills trust is simpler: the agent references code that no longer exists. A function signature changed two hours ago, and the model confidently suggests calling it with the old arguments. The edit compiles locally but breaks in CI because the model didn't know about a new required field.

Staleness compounds when agents make multi-step plans. If the first step is based on outdated context, every subsequent step is built on a false premise. The model can't self-correct because it doesn't know its initial assumptions were wrong.

Freshness requirements depend on your workflow. If you're using AI for one-off refactors in a stable codebase, indexing once per deploy might be fine. If you're running an agent loop that plans, edits, and validates across active feature branches, you need indexes that refresh on every meaningful commit—or you need to scope the agent to only the diff it's currently working on.

The three-tier index architecture that balances cost and latency

A single monolithic index updated on every push is expensive and slow. The pattern that works is layering three indexes with different refresh cadences:

**Tier 1: Structural metadata (updated on every commit).** This is your AST exports, function signatures, type definitions, and module boundaries. It's small enough to regenerate in seconds and critical enough that staleness here breaks everything. For a typical 50k-line TypeScript codebase, this might be 2–5 MB of JSON. You parse it, serialize it, and store it keyed by commit SHA.

**Tier 2: Semantic embeddings (updated on meaningful file changes).** This is your vector index for fuzzy search and similarity matching. It's slower to generate but doesn't need to be re-run when only comments change or when a variable is renamed. You can trigger this tier when diffs include actual logic changes, not just formatting.

**Tier 3: Dependency and call graphs (updated on release branches or nightly).** Full inter-module dependency resolution and call-graph analysis is expensive. You don't need it fresh for every feature branch. Run this on your main branch and periodically on long-lived branches.

The key insight is that tier 1 is your safety net. If the agent has fresh structural metadata, it won't hallucinate function signatures even if the semantic embeddings are a day old.

Incremental indexing: only re-parse what changed

Full-repo indexing on every commit doesn't scale past a few thousand files. The alternative is incremental indexing: track which files changed in the current commit, re-parse only those files, and merge the results into your existing index.

This requires two pieces of state:

1. A map from file path to its last-indexed commit SHA

2. The current index keyed by file path (or module name, or symbol)

On a new commit, you diff the tree against the last indexed state, re-run your parser on changed files, and update the index entries for those paths. The trade-off is complexity: you now have to handle file deletions, renames, and ensure your index stays consistent if a build is interrupted mid-update.

For Goatfied's agent loop, we index on every validation step. We parse the files the agent just edited, merge those into the working index, and pass the updated context into the next planning round. This keeps the agent's view of the codebase synchronized with the diffs it's actively making—critical when you're doing plan -> constrain -> edit -> validate cycles on a single feature.

Scoped context windows: index the PR, not the monorepo

If you're working in a large repository, indexing everything is both slow and wasteful. The agent doesn't need full context on every service in the monorepo—it needs the modules it's actively changing plus their immediate dependencies.

The pattern here is scoped indexing:

1. Identify the files in the current changeset (the PR diff or the agent's working set)

2. Walk their import/dependency tree one or two levels deep

3. Index only that subgraph

For a typical feature branch touching 10 files in a microservices monorepo, this might mean indexing 50–100 files instead of 5,000. The context is fresh, the index builds in seconds, and the agent isn't distracted by unrelated code.

The constraint-based planning in Goatfied's loop naturally supports this. When the agent makes a plan, we identify which modules it intends to touch, index those modules plus their direct dependencies, and use that as the working context. If the agent tries to reference something outside that scope, the validation step catches it before the edit is even attempted.

Handling schema evolution and backwards compatibility

Your index schema will change. You'll add new fields, rename keys, or switch from one AST parser to another. If you're storing indexes keyed by commit SHA, you need a strategy for handling old indexes that don't match your current schema.

The simplest approach: version your index format and include the version in the key. When you query for an index, check the version. If it's outdated, re-generate on the fly and cache the new version. This avoids a costly backfill while ensuring you can always reconstruct the index from source.

For Goatfied's self-hosted deployments, we write indexes to a local cache directory with a version prefix. When the editor updates and the index version bumps, we lazily regenerate as files are opened. Managed instances use the same pattern but store indexes in object storage, keyed by `<repo>/<commit>/<version>`.

Operational tips: what to monitor and when to rebuild

You need telemetry on index freshness, not just coverage. Track:

• **Lag between commit and index availability.** If this spikes, your indexing is falling behind.
• **Cache hit rate on index lookups.** Low hit rates mean you're re-indexing too often or your invalidation logic is broken.
• **Parse errors per file.** Syntax changes, new language features, or dependency updates can break your parser. You want to know immediately.

Set a staleness threshold: if the index for the current branch is more than N commits behind HEAD, trigger a full rebuild. For active branches, N might be 5. For release branches, N might be 50.

And always provide a manual rebuild escape hatch. When something breaks, the fastest fix is often "just reindex everything."

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)