Repository indexing that keeps context fresh

Stale context ruins more AI code edits than bad prompts. An agent confidently imports a function that was deleted three commits ago, or references an API contract that changed yesterday, and suddenly your compile-first loop is catching preventable errors instead of subtle logic bugs. The hard part isn't indexing a repository once—it's keeping that index accurate as the codebase shifts under you, without grinding your pipeline to a halt or teaching the LLM to hallucinate plausible-looking nonsense.

Production systems need indexing that can surface updated context within seconds for hot paths, tolerate slight lag for rarely-touched corners, and scale across repositories measured in tens of thousands of files. This guide walks through the design decisions, implementation patterns, and operational failures you'll hit building repository indexing that actually works when engineers are shipping.

What goes stale and why it breaks edits

File content is the obvious candidate—someone renames a class, adds a parameter, or deletes a helper. But dependency graphs go stale too: your agent thinks `UserService` imports `EmailValidator` because that was true at index time, but a recent refactor swapped in `NotificationClient`. Symbol tables, type definitions, test coverage maps, documentation embeddings—all of these decay the moment a developer pushes.

The symptom is confident hallucination. An LLM generates a call to `get_user(id: int)` when the real signature is now `get_user(user_id: UUID, tenant: str)`. Your compile gate catches it, the agent retries with the old context again, and you burn tokens in a loop. Fresh context means the agent sees the new signature before it writes the first line.

Staleness compounds when agents make multi-step plans. If the first step is based on outdated context, every subsequent step is built on a false premise. The model can't self-correct because it doesn't know its initial assumptions were wrong. In Goatfied's plan → constrain → edit → validate → retry loop, stale context during the plan phase poisons every downstream step.

Track structural changes, not just file diffs

Traditional file-watching approaches trigger re-indexing on save, but they miss the critical question: *what changed structurally*? A one-line type signature update can invalidate references across dozens of files, while a 200-line comment addition changes nothing that matters for code generation.

Effective indexing parses each changed file into an abstract syntax tree and diffs the structural elements:

• Exported functions, classes, and types
• Import/require statements and their resolved paths
• Public API surfaces vs. internal implementation
• Control flow patterns (early returns, error paths, async boundaries)

When `src/auth/session.ts` changes its exported `validateToken` signature from `(token: string)` to `(token: string, options?: TokenOptions)`, your index should immediately flag every call site, not just record that the file was modified. This structural diff determines which files need review before proposing an edit—if the agent sees a breaking change in the index, it can constrain its edits to account for affected call sites.

Not all code changes invalidate all context. A typo fix in a README doesn't require reindexing your entire type system, but adding a new required parameter to a widely-used function does. Categorize your indexed artifacts:

• **Type signatures and interfaces**: invalidate on AST changes to declarations, not implementation details
• **Dependency graphs**: invalidate when imports change or module boundaries shift
• **Test coverage maps**: invalidate only when test files or their direct dependencies change
• **Documentation embeddings**: invalidate on docstring edits, but batch updates to reduce reindex churn

Incremental vs full reindex: the tradeoff matrix

Full reindexing is simple: on every commit, walk the entire repository, parse every file, rebuild the symbol graph. It guarantees consistency—your index and working tree always match. The cost is latency and compute. A 10k-file Python monorepo might take 30–60 seconds to reparse with a tree-sitter pass, extract cross-references, and serialize embeddings. That's tolerable for nightly batch jobs; it's a dealbreaker if a developer is waiting to run an agent task.

Incremental indexing tracks changed files and updates only affected nodes. Parse the modified file, diff its exports and imports, ripple updates through the dependency graph. Done well, you can re-index a single-file change in under a second. For a typical PR touching 3-8 files, incremental updates complete in under 500ms, fast enough to stay in sync with LSP hover requests and AI context assembly.

The tradeoff is complexity: you need a dependency graph that supports partial updates, cache invalidation rules that don't leak stale edges, and careful handling of moves/renames that look like a delete + add. Track reverse dependencies explicitly:

const dependencyGraph = new Map<string, Set<string>>();



function indexFile(filepath: string) {

  const ast = parse(readFileSync(filepath));

  const imports = extractImports(ast);

  

  imports.forEach(imp => {

    if (!dependencyGraph.has(imp)) {

      dependencyGraph.set(imp, new Set());

    }

    dependencyGraph.get(imp).add(filepath);

  });

  

  updateIndex(filepath, ast);

}



function onFileChange(filepath: string) {

  indexFile(filepath);

  

  // Reindex files that import this one

  const dependents = dependencyGraph.get(filepath) || new Set();

  dependents.forEach(dep => indexFile(dep));

}

When `utils/format.ts` changes, you only re-parse files that import it, not the entire `/components` tree. For deep dependency trees, add a maximum cascade depth or a timeout. If a change affects more than 50 files, switch to a full rebuild. This prevents pathological cases where a change to a utility type triggers thousands of reindexes.

Hybrid strategies split the difference. Index changed files incrementally for interactive tasks, then run full reindexing overnight or weekly to catch drift. This works if you can detect when incremental updates have accumulated enough errors to matter—cycle detection in the dep graph, or a checksum mismatch between the incremental and a sampled full pass.

Triggering updates: Git hooks, file watchers, and webhooks

File system watchers (inotify, FSEvents) react instantly to edits, but they fire on every save, including half-written code that won't parse. Most systems throttle or batch watcher events—collect changes for 1–2 seconds, then trigger an index job. This avoids thrashing but adds latency.

Git hooks offer cleaner semantics. A post-commit hook knows the working tree is in a consistent state and gives you a diff between `HEAD` and `HEAD~1`:

#!/bin/bash

changed_files=$(git diff-tree --no-commit-id --name-only -r HEAD)

if echo "$changed_files" | grep -qE '\.(ts|tsx|py|go)$'; then

  goatfied index incremental --files "$changed_files"

fi

This hook indexes only when source files change, skipping commits that touch only docs or configs. The downside is coverage: hooks only fire on local commits. If a developer pulls changes from a remote branch, you need a separate `post-merge` or `post-checkout` hook. Goatfied's self-hosted runner uses `post-receive` hooks on the remote to queue indexing jobs as soon as a push lands, ensuring the central index is fresh before any agent task pulls context.

For CI environments or managed platforms, webhook-triggered indexing is more reliable. GitHub sends a `push` event; your indexer fetches the diff via API, processes changed files, and updates the index:

on:

  pull_request:

    types: [opened, synchronize]

jobs:

  index:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v3

        with:

          fetch-depth: 2

      - name: Index changed files

        run: |

          goatfied index branch ${{ github.head_ref }} \

            --base ${{ github.base_ref }}

Latency depends on webhook delivery and API rate limits—typically 1–5 seconds end-to-end for small pushes.

Detect staleness with Git metadata, not mtime

The naive approach polls file modification times and re-indexes anything with a newer timestamp than the last run. This breaks the moment a developer runs `git reset --hard` or switches branches: the index thinks everything is fresh because `mtime` hasn't changed, but the content is completely different.

Use `git log --name-status --format="%H %ct" <since>..<until>` to enumerate changed files between two commits. Parse the output to extract paths, then invalidate and re-index chunks that overlap those paths. Track the last-indexed commit SHA in metadata and diff from there to `HEAD`.

This approach handles branches, rebases, and force-pushes correctly because Git's object model is immutable. A chunk tagged with commit `abc123` is either still valid (that commit is in the ancestry of `HEAD`) or obsolete (it isn't). You can verify ancestry with `git merge-base --is-ancestor <chunk_sha> HEAD`; if the command fails, delete the chunk.

Store metadata alongside embeddings or structural indices: file path, start and end line numbers, commit SHA, and last-modified timestamp. The commit SHA is non-negotiable—without it, you can't tell whether a chunk represents the current HEAD or a branch from two weeks ago.

Handling branch-aware context

Teams working on multiple feature branches see different repository states. An agent on branch A needs context from A's index, not main's. But maintaining a full index per branch is expensive. Use a layered approach: a base index for main, plus delta indexes for each branch.

When an agent queries on branch B, merge the base index with B's delta. Changes in B override base entries:

function queryContext(branch: string, query: string): Context {

  const baseIndex = loadIndex('main');

  const branchDelta = loadIndex(branch);

  

  // Merge, with branch deltas taking precedence

  const merged = { ...baseIndex, ...branchDelta };

  return search(merged, query);

}

For PR workflows, index the diff and merge it into a branch-specific index. This keeps PR review context fresh without polluting the main index with experimental code. Delete branch deltas when the branch merges or is deleted. Set a TTL (say, 30 days) for abandoned branches to prevent unbounded delta growth.

Switching from `feature/api-redesign` to `main` can mean 30+ files appear, disappear, or change signatures. Hook into Git operations and maintain branch-aware state:

• Tag index snapshots by commit SHA or branch ref
• On checkout, diff the working tree against the cached index for that ref
• Apply incremental updates only to the changed subset
• Keep a short LRU cache of recent branch indices (main, develop, last 2-3 feature branches)

When you switch branches, the index loads the closest cached snapshot and patches forward instead of starting from zero.

Embedding drift and versioned namespaces

If you embed code snippets for semantic search, the model itself can go stale. A repository indexed six months ago with `text-embedding-ada-002` won't have the same vector space as fresh content embedded with a newer model or fine-tuned variant. Search quality degrades silently—queries return vaguely related chunks instead of the exact function the agent needs.

Most teams handle this with versioned embedding namespaces: `repo:myapp/v1/ada002` vs `repo:myapp/v2/ada003`. When you upgrade the embedding model, kick off a full reindex into a new namespace, run dual searches during a migration window to validate quality, then cut over. The cost is storage (you keep two copies briefly) and reindexing time.

A lighter-weight approach is incremental re-embedding: on each file change, regenerate embeddings only for that file and its immediate call graph. Stale embeddings linger in cold paths but hot paths stay current. Set a TTL on embeddings—anything older than 30 days gets queued for background refresh.

Track tokens sent to the embedding API per run, multiply by your provider's per-token cost, and set a budget alert. A 100K-line repository typically generates 5–10M tokens across all chunks; at $0.02 per million input tokens, that's $0.10–$0.20 for a full rebuild. Incremental updates should cost 1–5% of that per run.

Validation gates: how to know your index isn't lying

You can't trust an index just because it runs fast. Spot-check correctness by sampling random files and comparing index state to ground truth. If the index says `FooService` exports `bar()`, parse the file directly and confirm the export exists. Run this check asynchronously—every few hours or after every Nth incremental update—and alert if drift crosses a threshold (say, 1% of sampled entries mismatch).

Add a retrieval quality suite to CI: a set of queries with known-good file paths or function names that should appear in the top 5 results. Example test case: query "stripe webhook signature validation", assert that `src/payments/webhooks.rs:verify_signature` ranks in the top 3 chunks. Run this suite against a snapshot of your repository after every indexing change. Store test cases as JSON with the query, expected file paths, and acceptable rank thresholds.

Another validation is compiler alignment. Before an agent proposes a change, Goatfied's loop lints and type-checks with the repository's actual toolchain. If the compile step catches an undefined symbol the index swore was valid, that's a signal to re-index the relevant files. We log these mismatches and use them to tune invalidation heuristics—maybe moved files aren't triggering cascading updates the way they should.

In Goatfied's validate phase, we check proposed diffs against the index's type graph—if the agent tries to call `parseConfig(data, strict: true)` but the index shows `parseConfig` only accepts one argument, validation fails before the code even hits the compiler. This costs extra latency but prevents the much higher cost of bad commits. A failed compile from stale context can cascade into multiple retry cycles, each burning tokens and developer time.

Concurrent edits without index corruption

In AI-native workflows, the editing loop runs continuously: the agent proposes changes, gates validate them, failures trigger retries with adjusted constraints. If your index locks during updates, the entire loop stalls. If it doesn't lock, concurrent writes can corrupt dependency graphs.

A workable approach:

• Use a copy-on-write index structure (persistent data structures or snapshot-based stores)
• Read operations always see a consistent snapshot
• Write operations create a new snapshot and atomically swap the pointer
• Queue structural updates and batch them every 100-300ms

This lets validation gates read a stable index while the agent's next edit is already being parsed in the background.

Scoped context windows: index the PR, not the monorepo

If you're working in a large repository, indexing everything is both slow and wasteful. The agent doesn't need full context on every service in the monorepo—it needs the modules it's actively changing plus their immediate dependencies.

The pattern here is scoped indexing:

1. Identify the files in the current changeset (the PR diff or the agent's working set)

2. Walk their import/dependency tree one or two levels deep

3. Index only that subgraph

For a typical feature branch touching 10 files in a microservices monorepo, this might mean indexing 50–100 files instead of 5,000. The context is fresh, the index builds in seconds, and the agent isn't distracted by unrelated code.

When the agent makes a plan, identify which modules it intends to touch, index those modules plus their direct dependencies, and use that as the working context. If the agent tries to reference something outside that scope, the validation step catches it before the edit is even attempted.

Scaling indexing across many repositories

Single-repo indexing is table stakes. The hard problem is multi-tenancy: 500 projects, each with its own branch structure, language mix, and update cadence. Naive approaches queue every repo update into a single indexing worker, and the queue backs up during peak push hours.

Partition by repository or by tenant, and run parallel indexing workers. Each worker pulls jobs from its partition, so a giant monorepo re-indexing doesn't block a tiny microservice. Use priority queues to promote interactive tasks—if a developer is waiting on an agent, that index job jumps ahead of nightly batch work. Goatfied's managed offering does this with per-customer indexing clusters: your team's pushes never wait behind another team's backlog.

For very large monorepos, shard by directory or module. Index `/backend` and `/frontend` in parallel, then merge dependency graphs. This requires careful stitching—cross-boundary imports must resolve correctly—but it cuts indexing time roughly in half for 20k+ file repositories.

Operational monitoring and escape hatches

Track metrics that directly impact agent output quality:

• Median time from file save to index update completion (target: <500ms)
• Percentage of AI generations that reference outdated symbols (target: <2%)
• Index rebuild frequency per session (high frequency suggests inefficient incrementality)
• Log p95 indexing latency alongside chunk counts
• Parse errors per file (syntax changes or dependency updates can break your parser)

Set a staleness threshold: if the index for the current branch is more than N commits behind HEAD, trigger a full rebuild. For active branches, N might be 5. For release branches, N might be 50.

Always provide a manual rebuild escape hatch. When something breaks, the fastest fix is often "just reindex everything."

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)