Embedding models for code search: a practical comparison

When you need to find the function that handles OAuth token refresh across a hundred-thousand-line codebase, regex and grep fall short. Semantic code search—powered by embedding models—promises to surface relevant snippets even when you can't recall the exact variable names. But the landscape of code-specific embedding models has grown crowded, and each architecture makes different tradeoffs between speed, accuracy, and operational complexity.

We tested five embedding approaches on real-world retrieval tasks: finding functions by natural-language description, locating implementation patterns, and surfacing relevant context for code generation. The results matter because your choice of embedding model shapes both the latency of every search query and the relevance of what your AI assistant sees when it reaches into your codebase.

What makes code embeddings different from text

General-purpose sentence transformers like all-MiniLM-L6-v2 work surprisingly well on documentation, but code has properties that benefit from specialized training. Identifiers carry semantic weight—validate_auth_token tells you more than the function body might. Syntax structure matters: indentation, brackets, and call hierarchies encode relationships that bag-of-words models miss. And code often mixes natural language (comments, docstrings) with symbolic tokens in ways that trip up models trained purely on prose.

Code-specific models typically train on pairs like (function signature, docstring) or (issue description, implementing commit). They learn that def calculate_discount(price, coupon) should sit close to "apply promotional code to order total" in embedding space, even though the lexical overlap is minimal. The best models also handle cross-language patterns—recognizing that a Python function and a Go function solving the same problem should cluster together.

Models we tested

We compared five approaches across three repo sizes (10K, 100K, and 500K lines):

StarEncoder (Hugging Face, 2023): a 110M-parameter encoder trained on The Stack dataset. Supports 80+ languages and produces 768-dimensional embeddings. Inference runs on CPU without heroic optimization.

CodeBERT (Microsoft): one of the earliest code-specific transformers, pre-trained on docstring-code pairs from GitHub. Still widely used, 125M parameters, proven reliability.

UniXcoder (Microsoft, 2022): extends CodeBERT with unified cross-modal pre-training, so it understands comments and code as joint context rather than separate streams. Same 125M size but often sharper on semantic similarity.

OpenAI text-embedding-3-small: their general-purpose embedding API, 1536 dimensions. Not code-specific, but benefits from massive scale and continuous improvement. Requires API calls, so latency and cost differ from self-hosted options.

Voyage Code-2 (Voyage AI): a purpose-built code retrieval model, 1024 dimensions, accessed via API. Explicitly optimized for retrieval tasks, with reported gains on code search benchmarks.

We embedded chunks of 200–300 tokens (roughly one function or class) and measured two things: retrieval precision@5 (how often the correct snippet appears in the top five results for a natural-language query) and 95th-percentile latency for embedding + cosine-similarity search.

Retrieval precision: where specialization pays off

On the 100K-line Python/TypeScript monorepo, natural-language queries like "find the function that validates JWT expiry" surfaced the right snippet in the top five 74% of the time with UniXcoder, compared to 68% for CodeBERT and 61% for OpenAI's general model. StarEncoder landed at 71%. Voyage Code-2 hit 76%, the highest in our tests, but with API round-trip overhead.

The gap widened when queries mixed implementation details: "pagination logic that uses cursor tokens" or "retry decorator with exponential backoff." Code-specific models better understood that @retry(backoff=2) and "exponential backoff decorator" describe the same artifact. OpenAI's model occasionally surfaced documentation mentioning "retry" but missed the actual decorator implementation.

Cross-language retrieval told a different story. When asked to find "functions that parse ISO 8601 timestamps" in a polyglot repo (Python, Go, Rust), Voyage Code-2 and StarEncoder correctly grouped similar logic across languages. CodeBERT, trained primarily on Python and Java pairs, struggled with Rust syntax. If your codebase spans many ecosystems, training corpus breadth matters as much as architecture depth.

Latency and operational cost

Self-hosted models win on predictable latency. StarEncoder on a modest 4-CPU instance embeds a 250-token chunk in ~40ms (p95), and cosine search over 50K embeddings adds another 10ms with a basic FAISS index. CodeBERT and UniXcoder clock similar numbers. For interactive search in an editor—where every 100ms of lag is felt—this responsiveness matters.

API-based models introduce network round-trips. OpenAI text-embedding-3-small averaged 120ms p95 in our region, including TLS handshake and queuing. Voyage Code-2 was slightly faster at 95ms but still triple the self-hosted baseline. If you're embedding thousands of chunks during index builds, those milliseconds compound. A 100K-line codebase might contain 15,000 chunk-worthy snippets; at 100ms per call, that's 25 minutes of wall-clock time (parallelizable, but still a consideration for CI pipelines that rebuild indexes on every merge).

Cost structure also diverges. OpenAI charges per million tokens; embedding 15,000 chunks at ~250 tokens each costs roughly $0.50 per full reindex. Voyage pricing is similar. Self-hosted models pay only for compute: a continuously-running t3.medium instance costs ~$30/month and handles thousands of embedding requests per minute. If you rebuild indexes daily or offer search to every developer, the break-even point arrives quickly.

When general-purpose embeddings are enough

OpenAI's non-specialized model still delivered 61% precision@5, which might suffice if your codebase is well-commented and queries are high-level ("authentication logic" rather than "JWT expiry validation"). The simplicity of a single API call—no model hosting, no versioning—has value, especially for prototypes or small teams without ML infrastructure.

We also found that very large codebases (500K+ lines) benefited less from model choice than from chunk strategy. A mediocre embedding model with smart chunking—splitting at logical boundaries, preserving function signatures and leading comments—outperformed a better model with naive line-count splits. The retrieval problem shifts from "encode semantics well" to "define what a retrievable unit even is."

Integrating embeddings into agent workflows

At Goatfied, code search embeddings feed the planning phase of the agent loop. When the LLM receives a high-level task like "add rate limiting to the API," it issues semantic queries—"existing middleware patterns," "request throttling examples"—to find relevant context before generating diffs. The constraint that embeddings must refresh quickly matters here: if a developer just committed a new RateLimiter class, the agent should see it within seconds, not after the next nightly reindex.

We batch-embed changed files on every commit, then upsert those vectors into a running FAISS index. StarEncoder's 40ms embedding latency keeps this incremental update lightweight. The compile/lint/test gates that follow the edit phase catch when the agent misunderstood the retrieved context—validation failures trigger a retry with refined search terms or expanded snippets.

Small reversible diffs also reduce the blast radius of retrieval mistakes. If the agent pulls the wrong example and generates a flawed implementation, the diff is narrow enough to revert or debug quickly. This is harder when embeddings surface sprawling context and the agent produces 500-line changes.

Choosing your embedding stack

If you need cross-language coverage and can self-host, StarEncoder offers the best balance of breadth, speed, and simplicity. It handles polyglot repos well and runs on modest hardware.

If maximum retrieval precision on Python/TypeScript justifies API calls, Voyage Code-2 edges ahead, though you'll need to architect around latency and batching to keep costs reasonable.

If you're prototyping or have a small codebase, OpenAI's general embedding model is the fastest path to "good enough," especially if you already use their LLM APIs.

If you want to fine-tune on your own codebase—learning that your team's session_manager.py is semantically close to "user authentication" even though you never use that phrase—UniXcoder or CodeBERT provide solid starting checkpoints. Expect to invest in a training pipeline and labeled query-snippet pairs.

Related reading

• When fine-tuning beats RAG for code, and when it doesn't
• Quantization tradeoffs for code models: what you actually lose