Repository Indexing That Keeps Context Fresh Operational Checklist: practical systems guide

Your AI coding assistant hallucinates a function that was deleted three commits ago. The retrieval layer surfaces stale imports from a refactored module. A well-intentioned change silently breaks because the agent's mental model is two days out of date. Repository indexing isn't an implementation detail—it's the difference between an assistant that helps and one that ships broken code.

Fresh context requires operational rigor. The code lives in Git, the index lives in a vector store, and entropy guarantees they'll drift unless you actively synchronize them. Here's how to build and maintain a repository indexing pipeline that actually works.

Build the baseline indexing pipeline first

Start with a single-shot index builder that walks the repository, chunks every relevant file, and writes embeddings to your vector store. The simplest version uses a recursive directory walker filtered by `.gitignore` rules, breaks files at function boundaries or fixed line counts (typically 50–100 lines with 10-line overlap), and generates embeddings via your model provider's batch API.

Ignore performance for the first pass. Run it locally against a 10K-line repo, verify you can retrieve the chunks that matter, then measure cold-start time. Most small-to-mid-sized repositories finish in under 60 seconds on commodity hardware; if yours takes longer, profile before optimizing. Common bottlenecks: synchronous API calls to the embedding provider (batch them), reading large binary files (filter by extension early), and writing to a remote vector store without connection pooling.

Store metadata alongside the embedding: file path, start and end line numbers, commit SHA, and last-modified timestamp. You'll need these fields later to decide which chunks are stale. The commit SHA is non-negotiable—without it, you can't tell whether a chunk represents the current HEAD or a branch from two weeks ago.

Detect staleness with Git metadata, not mtime

The naive approach polls file modification times and re-indexes anything with a newer timestamp than the last run. This breaks the moment a developer runs `git reset --hard` or switches branches: the index thinks everything is fresh because `mtime` hasn't changed, but the content is completely different.

Use `git log --name-status --format="%H %ct" <since>..<until>` to enumerate changed files between two commits. Parse the output to extract paths, then invalidate and re-index chunks that overlap those paths. If your indexing job runs every five minutes, track the last-indexed commit SHA in the vector store's metadata table and diff from there to `HEAD`.

This approach handles branches, rebases, and force-pushes correctly because Git's object model is immutable. A chunk tagged with commit `abc123` is either still valid (that commit is in the ancestry of `HEAD`) or obsolete (it isn't). You can verify ancestry with `git merge-base --is-ancestor <chunk_sha> HEAD`; if the command fails, delete the chunk.

Incremental updates beat full re-indexing

Once the index exceeds a few thousand chunks, full rebuilds waste time and money. Incremental updates keep the round-trip under 10 seconds: identify changed files, delete obsolete chunks, re-chunk the new content, generate embeddings for the delta, and upsert.

Structure your vector store queries to support efficient deletes by file path. If your store supports metadata filtering, `DELETE WHERE file_path = 'src/api/users.rs'` removes every chunk from that file in one operation. Otherwise, maintain a secondary lookup table mapping paths to chunk IDs and iterate through deletes—slower but still tractable at mid-scale.

Batch the embedding API calls. Most providers charge per token and rate-limit by requests per minute; sending 100 small requests is both slower and more expensive than one batched request with 100 inputs. Aim for batches of 50–200 chunks depending on your token limits.

Track per-file chunk counts. If a file previously had 8 chunks and now has 12, you've added 4 chunks; if it drops to 5, you've deleted 3. Log these deltas. When something goes wrong—retrieval misses obvious matches, the agent references deleted code—the chunk-count timeline tells you whether the index is out of sync.

Handle long-lived branches and PRs

Developers work on feature branches for days or weeks. If your index only tracks `main`, the agent won't see their in-progress changes when they ask for help. If you index every branch, you'll spend a fortune on storage and embeddings for work that never ships.

Index `main` continuously and index active feature branches on-demand. When a developer starts a chat session in a branch, trigger an incremental index for files changed since the merge base with `main`. Store these branch-specific chunks in a separate namespace or tag them with the branch name in metadata. When querying, search both the `main` index and the current branch's delta, then deduplicate by file path (preferring the branch version).

Expire branch indexes after the branch merges or goes stale. Run a nightly job that lists branches with no new commits in the past week, deletes their chunk namespaces, and logs the cleanup. This keeps storage costs predictable without manual intervention.

Validate retrieval quality in CI

Unit tests verify the indexing pipeline runs without errors; they don't prove the agent retrieves the right context. Add a retrieval quality suite to your CI: a set of queries with known-good file paths or function names that should appear in the top 5 results.

Example test case: query "stripe webhook signature validation", assert that `src/payments/webhooks.rs:verify_signature` ranks in the top 3 chunks. Run this suite against a snapshot of your repository after every indexing change. If precision drops, you've either broken chunking logic or introduced a regression in embedding generation.

Store test cases as JSON with the query, expected file paths, and acceptable rank thresholds. Start with 10–20 cases covering core modules, high-churn areas, and past retrieval failures. Grow the suite organically as developers report "the agent couldn't find X" issues—each becomes a test case to prevent regressions.

Monitor embedding cost and latency

Indexing isn't free. Track tokens sent to the embedding API per run, multiply by your provider's per-token cost, and set a budget alert. A 100K-line repository typically generates 5–10M tokens across all chunks; at $0.02 per million input tokens, that's $0.10–$0.20 for a full rebuild. Incremental updates should cost 1–5% of that per run.

Log p95 indexing latency alongside chunk counts. If latency spikes without a corresponding increase in changed files, you've hit a bottleneck: slow vector store writes, embedding API rate limits, or sequential processing that should be parallel. Profile and fix before developers notice slow assistant responses.

Related reading

• [Goatfied vs Cursor vs GitHub Copilot: a benchmark across 50 real PR tasks](/blog/goatfied-vs-cursor-vs-github-copilot-benchmark-50-pr-tasks)
• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)