Keeping repository context fresh without reindexing everything

Every time you add a new service, rename a module, or merge a refactor, your editor's understanding of the codebase drifts a little further from reality. Full reindexing works, but it's a blunt instrument: most projects only touch a handful of files per commit, yet traditional indexing systems churn through the entire repository tree, re-parsing tens of thousands of lines that haven't changed. The latency adds up, context suggestions lag behind HEAD, and engineers learn to ignore stale autocomplete or—worse—trust outdated function signatures.

Incremental repository indexing treats the repository as a living structure instead of a snapshot. When you commit a change, only the affected files and their immediate dependents get re-analyzed. The index stays current without the overhead of rebuilding everything from scratch, and your editor's understanding of the codebase remains aligned with the code you're actually running.

Why full reindexing doesn't scale with real development velocity

A typical microservices monorepo might contain 50,000 files. Parsing, building symbol tables, and resolving cross-file references for the entire tree takes minutes, even with parallelization. If your team merges changes every few minutes, you're constantly racing against staleness—either you reindex on every pull and accept the latency, or you reindex on a timer and accept drift.

The worst case is when you're working across a feature branch with ten commits. Each time you check out a commit to debug something, a full reindex locks up your language server. Engineers start avoiding branch switches or disabling indexing altogether, which defeats the point of having context-aware tooling in the first place.

Incremental indexing short-circuits this by tracking which files changed between commits. If you switch from main to feature/api-v2 and only auth_handler.py and schema.proto differ, the indexer invalidates just those entries and their direct dependents—no need to re-walk vendor/, parse test fixtures, or rebuild the type graph for unrelated modules.

Identifying what actually needs to be re-indexed

The core challenge is dependency tracking. When auth_handler.py changes, you need to invalidate its own index entry, plus anything that imports it, plus anything that imports those importers. Stop too early and stale references leak through. Go too far and you're back to full reindexing.

Most incremental systems rely on the language's import graph. If you maintain a reverse-dependency map—file A → [files that import A]—you can walk it to compute the invalidation set. For TypeScript, you'd parse import statements. For Go, you'd extract import directives. For Python, you'd handle both import and from ... import, accounting for relative imports.

# Simplified reverse-dep tracking in a Python indexer

dependencies = {}

for module in all_modules:

    for imported in module.imports:

        if imported not in dependencies:

            dependencies[imported] = set()

        dependencies[imported].add(module)



def invalidate(changed_file):

    queue = [changed_file]

    visited = set()

    while queue:

        current = queue.pop()

        if current in visited:

            continue

        visited.add(current)

        # Re-index current

        if current in dependencies:

            queue.extend(dependencies[current])

This works until you hit dynamic imports, generated code, or cross-language boundaries. A Protobuf schema change might invalidate both the generated Python stubs and the Go bindings. A Terraform module rename affects tfvars files that reference it by path. For these cases, you need explicit dependency hints—either through structured comments, manifest files, or heuristics like "if *.proto changes, invalidate *_pb2.py."

Handling renames and deletions without orphaning references

File moves and deletions are where naive incremental indexing breaks down. If you rename old_auth.py to new_auth.py, a simple diff shows old_auth.py deleted and new_auth.py added. If you only invalidate those two entries, you miss all the files that still import old_auth—they'll keep stale references until something forces a full reindex.

The solution is to treat renames as atomic operations. Git already tracks renames with similarity scoring (git diff --name-status shows R100 for exact renames). When your indexer sees a rename, it:

1. Updates the internal file-to-module mapping

2. Invalidates the new file location

3. Invalidates every file in the reverse-dependency set of the old location

For deletions, you explicitly mark the entry as removed and trigger revalidation of anything that referenced it. This way, the next time an importer is parsed, it surfaces a "module not found" error instead of silently using cached data.

Goatfied's agent loop benefits from this because the validation step runs lint and type checks after every edit. If a rename introduced a broken import, the linter catches it immediately, and the agent's next retry can fix it—no human debugging required.

Trading off granularity and invalidation cost

You can track dependencies at different levels: file-level, module-level, symbol-level, or even finer. Finer granularity means smaller invalidation sets, but higher bookkeeping overhead.

File-level is the sweet spot for most repositories. If database.py changes, you invalidate database.py and all files that import it. This occasionally over-invalidates—if you only add a private helper function, dependents don't strictly need reindexing—but the wasted work is usually negligible compared to the complexity of tracking intra-file dependencies.

Symbol-level tracking makes sense for very large files or languages with slow parsers. If you maintain a per-symbol dependency graph, changing a private function in a 5,000-line module only invalidates the handful of call sites, not the entire module and its importers. The downside is you need precise symbol resolution during indexing, which is expensive for languages like Python where names can be rebound at runtime.

In practice, most teams start with file-level tracking, measure invalidation overhead, and only move to symbol-level if profiling shows a bottleneck.

Incremental indexing in distributed and multi-developer setups

When you're the only developer, incremental indexing is straightforward: you track local file changes, invalidate as needed, and reindex on every commit. In a team setting, the index needs to stay synchronized across multiple machines and branches.

One approach is to version the index itself. After each commit, serialize the index state—symbol tables, dependency graphs, file checksums—and commit it to a hidden directory like .goatfied/index/. When a teammate pulls your branch, they fetch your index state and only reindex files that differ between their working tree and the committed index. This works if index serialization is fast and the serialized format is stable.

A cleaner model is to treat the index as a build artifact. Your CI system runs indexing on every merge to main, caches the result, and makes it available as a downloadable blob. Local editors download the closest index (by commit SHA), then apply incremental updates for uncommitted changes. This avoids polluting the repository with index state but requires infrastructure for artifact storage.

Goatfied's self-hosted and managed deployments both support this model: the platform indexes the repository on every push, stores the result in object storage, and serves it to connected editors. When you create a branch, the editor downloads the parent commit's index and applies incremental updates as you edit. This keeps latency low even for repositories with millions of lines, because you're only reindexing the files you actually changed.

Handling generated code and build-dependent artifacts

Generated files—Protobuf stubs, GraphQL schema types, ORM models—complicate incremental indexing because they don't exist until you run the build. If you edit schema.proto and the indexer tries to parse schema_pb2.py before regenerating it, you get stale or missing symbols.

The solution is to hook indexing into the build process. After each build step that generates code, the indexer scans the output directory, checksums the generated files, and invalidates any entries whose checksums changed. This ensures the index always reflects the post-build state, not the pre-build source tree.

For languages with separate compilation, you can do better by indexing intermediate representations. A C++ indexer might parse .o files to extract symbol tables instead of re-parsing headers. A Rust indexer might read rmeta files from the incremental compilation cache. This sidesteps re-running the full compiler for files that didn't change.

Goatfied's constraint system enforces that compile and lint steps pass before the agent considers an edit complete. This guarantees that generated code is always up to date when the validation step runs, so incremental indexing sees the correct post-build state.

When to fall back to full reindexing

Incremental indexing is a performance optimization, not a correctness guarantee. Sometimes you need to rebuild from scratch:

• Schema migrations: If your build system changes how it resolves dependencies, the reverse-dependency map might be wrong.
• Index version upgrades: When you update the indexer itself, the serialized index format might be incompatible.
• Corruption or drift: If the incremental logic has a bug, the index can diverge from ground truth over time.

The safest approach is to periodically verify the incremental index against a full rebuild. Run a nightly job that reindexes the repository from scratch and diffs the result against the incremental index. If the diff is non-empty, log the discrepancies and fall back to the full index. This catches bugs in the incremental logic before they propagate to developers.

Goatfied handles this by treating the index as a cache: if the system detects drift or corruption, it silently falls back to full reindexing in the background while continuing to serve stale results. This keeps the editor responsive even when something goes wrong.

Related reading

• Pull request descriptions engineers actually read
• Risk-scoring pull requests for smarter review triage