Building evaluation sets from your own codebase

Most teams building code-generation tooling face the same uncomfortable truth: public benchmarks like HumanEval tell you almost nothing about how a model will perform on your codebase. The function signatures are different, the idioms are different, the dependencies are different, and the edge cases that matter to your domain simply aren't represented. If you want to know whether a fine-tuned model, a new RAG pipeline, or a different prompting strategy actually improves suggestions for your team, you need evaluation examples drawn from the code your engineers actually write.

The challenge isn't just collecting code—it's curating examples that are realistic, diverse, and automatable as test cases. This post walks through how to extract, filter, and structure evaluation sets from your own repositories, what makes a good eval example, and how to run them in a way that catches regressions before your engineers do.

Why public benchmarks don't transfer

HumanEval and MBPP are useful for comparing base model capabilities, but they measure performance on isolated algorithmic puzzles with no context beyond a docstring. Real code generation happens in the presence of:

• Existing type definitions, interfaces, and schemas
• Project-specific utilities and patterns
• Framework idioms (Django ORM queries, React hooks, Terraform resources)
• Internal libraries with no public documentation
• Non-obvious edge cases from production incidents

A model that scores 85% on HumanEval might confidently generate an ORM query that triggers N+1 problems in your Django app, or use a deprecated API from your internal SDK. Custom evaluation sets let you measure what matters: does the model respect your conventions, avoid your known pitfalls, and produce code that would pass your CI gates?

Sourcing candidates from Git history

The richest source of realistic examples is your Git history. Every merged pull request represents code that passed human review, ran through your test suite, and solved a real problem. The trick is identifying self-contained units that can be turned into eval cases.

Start with commits that modify a single function or method. You want changes where:

• The before state provides clear context (existing function signature, surrounding code)
• The after state is the ground truth (what a good model should generate)
• The diff is small enough to evaluate (10–50 lines changed, not 500)

A simple filter using git log and diff analysis:

git log --pretty=format:"%H" --since="6 months ago" | while read commit; do

  files=$(git diff-tree --no-commit-id --name-only -r $commit | grep "\.py$")

  for file in $files; do

    added=$(git show $commit -- $file | grep "^+" | wc -l)

    if [ $added -gt 10 ] && [ $added -lt 50 ]; then

      echo "$commit $file $added"

    fi

  done

done

This gives you a list of commits and files worth inspecting. You'll still need manual curation—many commits refactor multiple functions or introduce new dependencies that make isolated evaluation impossible—but it narrows the search.

What makes a good eval example

Not every code change is a useful evaluation case. The best examples are self-contained, unambiguous, and representative of real engineering work.

Self-contained means the model can generate the target code given only the surrounding context in the file, without needing to read five other modules. If the change depends on a new import or a schema defined elsewhere, include that context explicitly in the prompt setup, or skip it.

Unambiguous means there's a clear correct answer. Refactors where two different implementations are both valid aren't good evals—you'll waste time debating whether the model "got it right." Look for cases where the change fixes a bug, implements a well-defined feature, or follows an established pattern.

Representative means the example reflects work your team does regularly. If you're evaluating a model for a Django codebase, prioritize view functions, ORM queries, and serializer logic over one-off data migration scripts.

Concretely, strong candidates include:

• Adding a new model method that follows existing patterns
• Implementing a missing error-handling branch
• Adapting an existing function to a new API version
• Writing a new test case for a regression

Weak candidates:

• Large refactors touching many functions
• Stylistic changes (renaming variables, reordering imports)
• Code that only makes sense with knowledge of a Slack conversation or ticket

Structuring the evaluation set

Once you've identified 50–100 good candidates, structure them in a consistent format so you can run them through different models or configurations and compare results automatically.

Each eval case needs three pieces:

1. Context: The file state before the change, plus any relevant imports or adjacent code

2. Instruction: A natural-language description of what to generate (ideally extracted from the commit message or PR description)

3. Ground truth: The actual code that was merged

Store these as JSON or JSONL:

{

  "id": "eval_042",

  "context": "class OrderSerializer(serializers.ModelSerializer):\n    ...",

  "instruction": "Add a custom validate method that checks inventory before confirming order",

  "ground_truth": "    def validate(self, attrs):\n        product = attrs['product']\n        if product.inventory < attrs['quantity']:\n            raise serializers.ValidationError('Insufficient inventory')\n        return attrs",

  "metadata": {

    "commit": "a3f29c1",

    "file": "api/serializers.py",

    "tags": ["django", "validation"]

  }

}

The metadata field is useful for stratifying results—you might want to know whether your model struggles more with Django ORM queries than with pure business logic, or whether it regresses on older files versus recently modified ones.

Running evaluations and defining pass criteria

The simplest pass criterion is exact match: does the generated code match the ground truth character-for-character? This is too strict for code—there are many valid ways to format braces, name variables, or order parameters—but it's a useful baseline. If your model gets exact matches on 30% of examples, you know it's at least capable of memorizing common patterns.

A more realistic criterion is functional equivalence: does the generated code pass the same tests and produce the same behavior? For evaluation sets extracted from Git history, you can:

1. Check out the commit before the change

2. Replace the old code with the model's generated code

3. Run your existing test suite

If tests that failed before the change now pass, the generation is functionally correct. If tests that passed before now fail, the generation introduced a regression.

This requires that your eval examples have corresponding test coverage, which isn't always the case. For examples without tests, you can fall back to:

• Compilation/linting: Does the generated code parse and pass your linters? This catches syntax errors and obvious type mismatches.
• Human review: Does a human engineer judge the generated code as "acceptable to merge"? Expensive to scale, but useful for spot-checking.

At Goatfied, we run evaluations in the same compile → lint → test loop that gates production edits. If generated code wouldn't pass CI, it fails the eval—no exceptions. This keeps the eval pipeline honest.

Measuring improvement, not just accuracy

Absolute accuracy matters less than relative improvement. If you're comparing two prompting strategies, or deciding whether fine-tuning is worth the effort, you want to know: does approach B do better than approach A on the code patterns that matter to your team?

Track metrics per tag or category:

• Pass rate on Django ORM queries: 42% → 58%
• Pass rate on error-handling branches: 31% → 29% (regression!)
• Pass rate on recent code (<3 months old): 55% → 61%

This granularity lets you make informed tradeoffs. Maybe a fine-tuned model does worse overall but significantly better on your most painful category (async TypeScript, Terraform modules, whatever). That might be worth shipping.

Also track edit distance or diff size for near-misses. If a model generates code that's 95% correct and needs only a small manual fix, that's far more valuable than code that's 50% correct and requires a full rewrite. Levenshtein distance on the final code or line-level diff metrics give you this signal.

Keeping the dataset fresh

Your codebase evolves. New frameworks get adopted, old patterns get deprecated, and the kinds of code your team writes shift over time. If your evaluation set is static, you'll eventually be measuring performance on code nobody writes anymore.

Set up a lightweight pipeline to refresh your dataset quarterly:

• Pull the last N months of commits
• Run your filtering heuristics
• Add new examples that represent emerging patterns (e.g., new API endpoints, new test styles)
• Archive examples that are no longer relevant (code for a deprecated service, outdated syntax)

Keep a frozen "benchmark" subset for longitudinal comparisons—this lets you track whether model performance is improving over time without the confound of a changing dataset—but continuously expand the working set.

Integration with existing CI

The real payoff from custom evaluation sets is catching regressions before you ship. If you're fine-tuning a model, adjusting retrieval weights, or switching to a new base model, run your eval suite as a CI check. If pass rate drops below a threshold, fail the build.

In Goatfied's self-hosted deployments, teams often wire their eval sets into the agent loop's validation step. When the agent proposes an edit, it first checks: would this edit pass the same functional criteria as our curated examples? If not, the agent retries or escalates. This creates a tight feedback loop where evaluation criteria directly shape generation behavior.

Related reading

• When fine-tuning beats RAG for code, and when it doesn't
• Quantization tradeoffs for code models: what you actually lose