Repository Indexing That Keeps Context Fresh Production Playbook: practical systems guide

Most AI coding assistants lose the plot around line 200. They'll happily suggest a refactor that breaks an import three files away, or generate a new function that duplicates logic already sitting in `utils/`, because their context window sampled the wrong 10% of your repository. The problem isn't model intelligence—it's that repository indexing treats code like static documentation instead of a living dependency graph that changes every commit.

Production repository indexing isn't about embedding every file and hoping semantic search finds the right chunks. It's about maintaining an up-to-date map of symbols, dependencies, and structural relationships that survives merges, branch switches, and concurrent edits. Here's how to build indexing that actually keeps context fresh when engineers are shipping.

Track structural changes, not just file diffs

Traditional file-watching approaches trigger re-indexing on save, but they miss the critical question: *what changed structurally*? A one-line type signature update can invalidate references across dozens of files, while a 200-line comment addition changes nothing that matters for code generation.

Effective indexing parses each changed file into an abstract syntax tree and diffs the structural elements:

• Exported functions, classes, and types
• Import/require statements and their resolved paths
• Public API surfaces vs. internal implementation
• Control flow patterns (early returns, error paths, async boundaries)

When `src/auth/session.ts` changes its exported `validateToken` signature from `(token: string)` to `(token: string, options?: TokenOptions)`, your index should immediately flag every call site, not just record that the file was modified. Goatfied's plan phase uses this structural diff to determine which files need review before proposing an edit—if the agent sees a breaking change in the index, it constrains its edits to account for affected call sites.

Incremental updates beat full re-scans

Rebuilding your entire index on every change creates a window where context is stale. In a 50K-line repository, full re-indexing might take 8-15 seconds with a TypeScript parser. That's long enough for an engineer to switch branches or merge main, making your "fresh" index immediately outdated.

Incremental indexing maintains a dependency graph and updates only affected nodes:

// Pseudocode for incremental update

function updateIndex(changedFiles: string[]) {

  const affectedNodes = new Set(changedFiles);

  

  for (const file of changedFiles) {

    const oldImports = index.getImports(file);

    const newImports = parseImports(readFile(file));

    

    // Files that imported this one may need re-analysis

    for (const importer of index.getImporters(file)) {

      affectedNodes.add(importer);

    }

    

    // Update reverse dependency map

    index.updateNode(file, newImports);

  }

  

  return reindexSubgraph(affectedNodes);

}

The key insight: when `utils/format.ts` changes, you only re-parse files that import it, not the entire `/components` tree. For a typical PR touching 3-8 files, incremental updates complete in under 500ms, fast enough to stay in sync with LSP hover requests and AI context assembly.

Version control awareness prevents phantom references

Switching from `feature/api-redesign` to `main` can mean 30+ files appear, disappear, or change signatures. Naive indexing either keeps stale references from the old branch or forces a full rebuild. Both approaches break: the first suggests code that doesn't exist, the second creates a gap where the index is rebuilding while an engineer is already prompting an AI assistant.

Effective indexing hooks into Git operations and maintains branch-aware state:

• Tag index snapshots by commit SHA or branch ref
• On checkout, diff the working tree against the cached index for that ref
• Apply incremental updates only to the changed subset
• Keep a short LRU cache of recent branch indices (main, develop, last 2-3 feature branches)

When you switch branches, the index loads the closest cached snapshot and patches forward instead of starting from zero. This matters especially for AI workflows where the engineer might jump between `main` to check current behavior and `feature/*` to implement changes—context should follow instantly.

Handle concurrent edits without index corruption

In AI-native workflows, the editing loop runs continuously: the agent proposes changes, gates validate them, failures trigger retries with adjusted constraints. If your index locks during updates, the entire loop stalls. If it doesn't lock, concurrent writes can corrupt dependency graphs.

A workable approach:

• Use a copy-on-write index structure (persistent data structures or snapshot-based stores)
• Read operations always see a consistent snapshot
• Write operations create a new snapshot and atomically swap the pointer
• Queue structural updates and batch them every 100-300ms

This lets validation gates read a stable index while the agent's next edit is already being parsed in the background. Goatfied's validate phase checks proposed diffs against the index's type graph—if the agent tries to call `parseConfig(data, strict: true)` but the index shows `parseConfig` only accepts one argument, validation fails before the code even hits the compiler.

Expose query APIs optimized for agent retrieval

LLMs don't want your full index—they want the minimal context slice needed for the current task. Design index queries that answer agent-specific questions:

• "What are all public exports from `@/lib/`?" → for import suggestions
• "Which files call `deprecated_authenticate`?" → for refactor scope
• "Show me error handling patterns in `services/`" → for consistent style
• "What changed in the last 3 commits to `schema/`?" → for migration awareness

These queries should return ranked results with enough surrounding context for the LLM to make decisions, but not so much that you blow the context budget. A typical response might include:

• The matched symbol definition (5-15 lines)
• Its immediate dependencies (imports, types)
• 2-3 representative usage examples from the same module
• Structural metadata (file path, export status, last modified timestamp)

Integrate with compile and lint gates

An index that doesn't reflect compilation reality is worse than no index—it teaches the AI assistant to generate code that looks right but fails the build. The most reliable approach is to rebuild relevant index segments after successful compile/lint passes, treating the compiler as the source of truth.

When Goatfied runs its edit → validate → retry loop, the validate step includes:

1. Incremental index update for changed files

2. Type check via `tsc --noEmit` or equivalent

3. Lint pass to catch style/pattern violations

4. If validation passes, finalize the index update

5. If validation fails, discard the index changes and retry with updated constraints

This ensures the index never drifts from what actually compiles. The agent learns from validated code only—failed attempts don't pollute the context.

Monitor index freshness as an operational metric

In production AI-assisted workflows, index staleness directly impacts output quality. Track:

• Median time from file save to index update completion (target: <500ms)
• Percentage of AI generations that reference outdated symbols (target: <2%)
• Index rebuild frequency per session (high frequency suggests inefficient incrementality)
• Memory footprint per 10K lines of code (watch for unbounded growth)

If your p95 index update time climbs above 1 second, engineers will experience lag between editing and getting accurate suggestions. If stale symbol references exceed 5%, you're shipping context that makes AI output actively misleading.

Repository indexing isn't glamorous infrastructure—it's the difference between an AI assistant that understands your codebase and one that hallucinates plausible-looking nonsense. Keep the index fresh, structure-aware, and synchronized with validation gates, and you build the foundation for agents that ship code that actually works.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)