Repository Indexing That Keeps Context Fresh Failure Patterns: practical systems guide

Stale repository indexes kill agent velocity. An LLM sees yesterday's function signature, generates a call with the wrong parameter count, and your CI fails. The agent retries with the same outdated context, fails again, and burns tokens in a loop. Fresh indexing sounds simple—reindex on every commit—but production systems reveal three failure modes that don't appear in demos: race conditions between writes and reads, cascade reindexes when monorepo dependencies shift, and silent drift when branches diverge faster than your indexer can reconcile them.

This guide walks through practical patterns for keeping repository context fresh without introducing new reliability holes. We'll cover incremental indexing strategies, when to invalidate vs. rebuild, and how to detect index staleness before it reaches your agent loop.

The staleness contract: defining "fresh enough"

Not all code changes invalidate all context. A typo fix in a README doesn't require reindexing your entire type system, but adding a new required parameter to a widely-used function does. The key is defining a staleness contract: which changes matter for which queries, and how much drift you tolerate before forcing a rebuild.

Start by categorizing your indexed artifacts:

• **Type signatures and interfaces**: invalidate on AST changes to declarations, not implementation details
• **Dependency graphs**: invalidate when imports change or module boundaries shift
• **Test coverage maps**: invalidate only when test files or their direct dependencies change
• **Documentation embeddings**: invalidate on docstring edits, but batch updates to reduce reindex churn

A simple staleness detector compares the last-indexed commit hash per file against HEAD. When the agent queries context, check if any relevant files have drifted. If drift exceeds your threshold (say, more than 5 files or any change in a core module), trigger a partial reindex before returning results. This trades a small latency hit for avoiding the much larger cost of agent retries on stale data.

Incremental indexing with event-driven triggers

Polling for changes doesn't scale. Use Git hooks or CI webhooks to trigger indexing on push, but filter events to avoid redundant work. A practical setup:

# .git/hooks/post-commit

#!/bin/bash

changed_files=$(git diff-tree --no-commit-id --name-only -r HEAD)

if echo "$changed_files" | grep -qE '\.(ts|tsx|py|go)$'; then

  goatfied index incremental --files "$changed_files"

fi

This hook indexes only when source files change, skipping commits that touch only docs or configs. The `incremental` mode updates the index for changed files without rebuilding the entire repository graph.

For team workflows, move the trigger to CI. On pull request update, index the diff and merge it into a branch-specific index. This keeps PR review context fresh without polluting the main index with experimental code:

# .github/workflows/index-pr.yml

on:

  pull_request:

    types: [opened, synchronize]

jobs:

  index:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v3

        with:

          fetch-depth: 2

      - name: Index changed files

        run: |

          goatfied index branch ${{ github.head_ref }} \

            --base ${{ github.base_ref }}

This workflow indexes the PR branch against its base, so agents working in that PR context see the proposed changes without waiting for a merge.

Handling dependency cascades in monorepos

Changing a shared type in a monorepo can ripple through dozens of packages. A naive incremental indexer updates the changed file but misses the cascade. Consumers still reference the old signature, and agents generate code that compiles locally but breaks integration.

Track reverse dependencies explicitly. When indexing a file, record which other files import it. On change, enqueue those dependents for reindex:

// Simplified dependency tracker

const dependencyGraph = new Map<string, Set<string>>();



function indexFile(filepath: string) {

  const ast = parse(readFileSync(filepath));

  const imports = extractImports(ast);

  

  imports.forEach(imp => {

    if (!dependencyGraph.has(imp)) {

      dependencyGraph.set(imp, new Set());

    }

    dependencyGraph.get(imp).add(filepath);

  });

  

  // Index this file's exports and types

  updateIndex(filepath, ast);

}



function onFileChange(filepath: string) {

  indexFile(filepath);

  

  // Reindex files that import this one

  const dependents = dependencyGraph.get(filepath) || new Set();

  dependents.forEach(dep => indexFile(dep));

}

For deep dependency trees, add a maximum cascade depth or a timeout. If a change affects more than 50 files, switch to a full rebuild. This prevents pathological cases where a change to a utility type triggers thousands of reindexes.

Reconciling divergent branch indexes

Teams working on multiple feature branches see different repository states. An agent on branch A needs context from A's index, not main's. But maintaining a full index per branch is expensive. Use a layered approach: a base index for main, plus delta indexes for each branch.

When an agent queries on branch B, merge the base index with B's delta. Changes in B override base entries. This keeps index size manageable and allows sharing the bulk of the context across branches:

function queryContext(branch: string, query: string): Context {

  const baseIndex = loadIndex('main');

  const branchDelta = loadIndex(branch);

  

  // Merge, with branch deltas taking precedence

  const merged = { ...baseIndex, ...branchDelta };

  return search(merged, query);

}

Delete branch deltas when the branch merges or is deleted. Set a TTL (say, 30 days) for abandoned branches to prevent unbounded delta growth.

Detecting silent drift with validation gates

The worst index failures are silent. The index looks fresh, the agent generates code, and it breaks in CI because the index missed a subtle change. Add validation gates:

• **Type-check generated code against the live repository**, not just the index. If they diverge, reindex and retry.
• **Compare index checksums on each agent run**. If the checksum changed mid-execution, abort and restart with fresh context.
• **Log index age per query**. If queries consistently hit indexes older than 5 minutes, your indexing cadence is too slow.

In Goatfied's agent loop, we validate generated diffs before applying them: `plan -> constrain -> edit -> validate -> retry`. The validate step compiles and lints against the actual repository, catching index drift before it reaches version control. If validation fails due to type mismatches, we reindex the affected files and retry the edit with updated context.

This costs extra latency but prevents the much higher cost of bad commits. A failed compile from stale context can cascade into multiple retry cycles, each burning tokens and developer time.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)