Repository Indexing That Keeps Context Fresh Design Tradeoffs: practical systems guide

Stale context ruins more AI code edits than bad prompts. An agent confidently imports a function that was deleted three commits ago, or references an API contract that changed yesterday, and suddenly your "compile-first" loop is catching preventable errors instead of subtle logic bugs. The hard part isn't indexing a repository once—it's keeping that index accurate as the codebase shifts under you, without grinding your pipeline to a halt.

Most repository indexing systems treat freshness as a binary: "reindex everything on push" or "accept whatever staleness batch jobs give you." Production systems need something in between—an architecture that can surface updated context to an LLM within seconds for hot paths, tolerate slight lag for rarely-touched corners, and scale across repositories measured in tens of thousands of files.

What actually goes stale and why it breaks edits

File content is the obvious candidate—someone renames a class, adds a parameter, or deletes a helper. But dependency graphs go stale too: your agent thinks `UserService` imports `EmailValidator` because that was true at index time, but a recent refactor swapped in `NotificationClient`. Symbol tables, type definitions, test coverage maps—all of these decay the moment a developer pushes.

The symptom is confident hallucination. An LLM generates a call to `get_user(id: int)` when the real signature is now `get_user(user_id: UUID, tenant: str)`. Your compile gate catches it, the agent retries with the old context again, and you burn tokens in a loop. Fresh context means the agent sees the new signature before it writes the first line.

Incremental vs full reindex: the tradeoff matrix

Full reindexing is simple: on every commit, walk the entire repository, parse every file, rebuild the symbol graph. It guarantees consistency—your index and working tree always match. The cost is latency and compute. A 10k-file Python monorepo might take 30–60 seconds to reparse with a tree-sitter pass, extract cross-references, and serialize embeddings. That's tolerable for nightly batch jobs; it's a dealbreaker if a developer is waiting to run an agent task.

Incremental indexing tracks changed files—usually by hooking Git's pre-commit or post-receive events—and updates only affected nodes. Parse the modified file, diff its exports and imports, ripple updates through the dependency graph. Done well, you can re-index a single-file change in under a second. The tradeoff is complexity: you need a dependency graph that supports partial updates, cache invalidation rules that don't leak stale edges, and careful handling of moves/renames that look like a delete + add.

Hybrid strategies split the difference. Index changed files incrementally for interactive tasks, then run full reindexing overnight or weekly to catch drift. This works if you can detect when incremental updates have accumulated enough errors to matter—cycle detection in the dep graph, or a checksum mismatch between the incremental and a sampled full pass.

File watchers and Git hooks: where to plug in updates

File system watchers (inotify, FSEvents) are tempting because they react instantly to edits. But they fire on every save, including half-written code that won't parse. In practice, most systems throttle or batch watcher events—collect changes for 1–2 seconds, then trigger an index job. This avoids thrashing but adds latency.

Git hooks offer cleaner semantics. A post-commit hook knows the working tree is in a consistent state and gives you a diff between `HEAD` and `HEAD~1`. You can extract exactly which files changed and which lines moved. The downside is coverage: hooks only fire on local commits. If a developer pulls changes from a remote branch, you need a separate `post-merge` or `post-checkout` hook. Goatfied's self-hosted runner uses `post-receive` hooks on the remote to queue indexing jobs as soon as a push lands, ensuring the central index is fresh before any agent task pulls context.

For CI environments or managed platforms, webhook-triggered indexing is more reliable. GitHub sends a `push` event; your indexer fetches the diff via API, processes changed files, and updates the index. Latency depends on webhook delivery and API rate limits—typically 1–5 seconds end-to-end for small pushes.

Embedding drift and when to regenerate vectors

If you embed code snippets for semantic search, the model itself can go stale. A repository indexed six months ago with `text-embedding-ada-002` won't have the same vector space as fresh content embedded with a newer model or fine-tuned variant. Search quality degrades silently—queries return vaguely related chunks instead of the exact function the agent needs.

Most teams handle this with versioned embedding namespaces: `repo:myapp/v1/ada002` vs `repo:myapp/v2/ada003`. When you upgrade the embedding model, kick off a full reindex into a new namespace, run dual searches during a migration window to validate quality, then cut over. The cost is storage (you keep two copies briefly) and reindexing time.

A lighter-weight approach is incremental re-embedding: on each file change, regenerate embeddings only for that file and its immediate call graph. Stale embeddings linger in cold paths but hot paths stay current. Set a TTL on embeddings—anything older than 30 days gets queued for background refresh.

Validation: how to know your index isn't lying

You can't trust an index just because it runs fast. Spot-check correctness by sampling random files and comparing index state to ground truth. If the index says `FooService` exports `bar()`, parse the file directly and confirm the export exists. Run this check asynchronously—every few hours or after every Nth incremental update—and alert if drift crosses a threshold (say, 1% of sampled entries mismatch).

Another validation is compiler alignment. Before an agent proposes a change, Goatfied's loop lints and type-checks with the repository's actual toolchain. If the compile step catches an undefined symbol the index swore was valid, that's a signal to re-index the relevant files. We log these mismatches and use them to tune invalidation heuristics—maybe moved files aren't triggering cascading updates the way they should.

Scaling indexing across many repositories

Single-repo indexing is table stakes. The hard problem is multi-tenancy: 500 projects, each with its own branch structure, language mix, and update cadence. Naive approaches queue every repo update into a single indexing worker, and the queue backs up during peak push hours.

Partition by repository or by tenant, and run parallel indexing workers. Each worker pulls jobs from its partition, so a giant monorepo re-indexing doesn't block a tiny microservice. Use priority queues to promote interactive tasks—if a developer is waiting on an agent, that index job jumps ahead of nightly batch work. Goatfied's managed offering does this with per-customer indexing clusters: your team's pushes never wait behind another team's backlog.

For very large monorepos, shard by directory or module. Index `/backend` and `/frontend` in parallel, then merge dependency graphs. This requires careful stitching—cross-boundary imports must resolve correctly—but it cuts indexing time roughly in half for 20k+ file repositories.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)