Prompt patterns for reliable test generation

Generating reliable tests with LLMs exposes the gap between "produces syntactically valid code" and "produces code that catches regressions." Most teams discover this when their AI-generated test suite passes perfectly on the first run—then fails to catch an obvious null-pointer bug three commits later. The problem isn't model capability; it's that test generation, treated as a one-shot prompt, produces technical debt faster than it produces confidence.

After iterating on thousands of test-generation loops in Goatfied's agent workflow—plan, constrain, edit, validate, retry—we've learned that reliability comes from treating prompts as contracts, not creative briefs. This guide walks through the architectural patterns and specific prompt structures that turn test generation from a risky convenience into a repeatable engineering practice.

Why test generation multiplies the failure surface

Most generative coding workflows treat the LLM as a one-shot compiler: feed in context, get back code, hope it works. Test generation multiplies the failure modes. You're asking the model to infer what to test, how to set up fixtures, which assertions express correctness, and when mocks versus integration boundaries make sense. Each of these is a judgment call with context dependencies—framework conventions, project style, existing test patterns.

The most insidious failure isn't a test that crashes. It's one that passes without actually verifying behavior. Models frequently generate assertions like `expect(result).toBeDefined()` or `assert response is not None`, which technically pass but prove nothing about correctness. A test that checks `expect(users.length).toBeGreaterThan(0)` looks reasonable until you realize it still passes when you comment out the actual filtering logic.

The brittle path is cramming all context into a single mega-prompt and praying. The durable path is decomposing test generation into constrained, validatable steps that survive partial failures and give you concrete feedback when something drifts.

Separate planning from implementation

The highest-leverage shift we made was splitting test generation into two phases: coverage planning and test implementation. The planning step produces a structured artifact—not code, but a test specification:

{

  "target": "src/pipeline/executor.ts",

  "cases": [

    {

      "name": "retries_on_transient_failure",

      "type": "integration",

      "spec": "Must retry with exponential backoff when DB returns transient error"

    },

    {

      "name": "respects_max_retry_limit", 

      "type": "unit",

      "spec": "Must fail after 3 retries, not loop indefinitely"

    }

  ]

}

This lets you validate coverage intent before writing a line of test code, regenerate implementations independently if the first attempt fails validation, and diff plans across runs to catch hallucinated test cases. The planning prompt is narrow: "Given this module and these existing tests, list missing coverage areas as a JSON array with concrete specifications." Implementation prompts then work case-by-case: "Write a test for `retries_on_transient_failure` that validates the spec."

In Goatfied's agent loop, this fits naturally into the plan phase. The spec extraction becomes part of the changeset context, and the constrain phase ensures generated tests compile against the correct interfaces before any files change. Two LLM calls instead of one, but planning is fast (low token count) and catches conceptual mistakes early—cheaper than expensive retries after generating broken code.

Anchor generation to the actual diff

The biggest mistake in test generation is asking the model to "write tests for this function" in isolation. You get back generic happy-path cases that would've passed before your change and will pass after it. Instead, anchor the prompt on the specific diff. Show the model what changed—new branches, refactored error handling, added edge cases—and ask it to generate tests that exercise those changes.

You are generating tests for this change:



{{ git diff }}



Focus on:

- New conditional branches introduced in lines 47-52

- The error case added for null paymentMethod  

- Changed return type from string to Result<string, PaymentError>



Generate test cases that would fail on the old implementation but pass on the new one.

This pattern produces tests directly tied to your work. If the diff adds a null check, the agent writes a test with a null input. If you refactor an exception into a Result type, it generates assertions for both Ok and Err variants. Vague prompts produce generic coverage; diff-anchored prompts produce targeted regression prevention.

Constraint-first prompts beat example-heavy prompts

Developers instinctively reach for few-shot examples—"here's a good test, write one like this." It works until the model overfits to the example's structure and produces tests that look right but assert nothing meaningful. We get better results from explicit constraints as negative requirements:

Write a test for `parseConfig` that:

- Uses the existing `mockFs` fixture (do not create a new filesystem mock)

- Asserts on the returned object shape, not implementation details  

- Does NOT test error messages verbatim (check error type only)

- Must fail if you comment out the validation logic



DO NOT:

❌ Mock the function under test itself

❌ Assert on private methods or internal state

❌ Check log output instead of return values

❌ Use sleeps or arbitrary timeouts

This forces the model to solve within boundaries rather than template-match. When generation fails validation—say, it invents a new mock—the retry prompt inherits the constraint and the error message, giving the model concrete correction signal.

Goatfied's validation gates (compile, lint, type-check) feed directly into the retry loop. A test that fails `tsc` gets re-prompted with the exact compiler error. A test that fails `eslint` gets the rule violation. This turns the prompt into a constraint solver with feedback, not a creative writer.

Anchor to existing test infrastructure

The fastest way to generate unusable tests is letting the LLM invent setup code from scratch. Every project has idioms—specific test runners, custom matchers, fixture factories. Ignoring these means tests that don't run or require manual surgery to integrate.

Give the model a concrete example test from your actual codebase before asking it to write new ones. Show it one passing test from the same module, then request coverage for a specific untested path:

// Include this in your prompt context:

describe('UserService', () => {

  it('creates user with valid email', async () => {

    const service = setupTestService();

    const result = await service.create({ email: 'test@example.com' });

    expect(result.id).toBeDefined();

    expect(result.email).toBe('test@example.com');

  });

});



// Then prompt: "Write a test following this structure for the password

// reset flow when the email doesn't exist in the database."

The model learns your test framework syntax, matches your team's assertion style, and inherits setup patterns (fixtures, mocking conventions) that would otherwise require extensive prompt engineering. When you ask for tests in a vacuum, models default to generic examples that rarely match your project's idioms.

We also extract infrastructure explicitly: parse the existing test directory to find common patterns (imports, setup hooks, helper functions), inject a schema of "available fixtures" into the generation context, and constrain the prompt: "Use `createTestUser()` from `fixtures/auth`, do not write your own user factory." This reduces creative surface area and ties generated code to the project's actual test harness. When we added this, our "test passes on first run" rate jumped from roughly 40% to 75% for typical CRUD modules.

Enforce diversity with coverage boundaries

LLMs love patterns. Show them one parameterized test and they'll generate twelve more with different input values but identical structure—except half will have off-by-one errors in the expected output or reuse the same mock setup where it doesn't apply. This copy-paste explosion wastes tokens and produces low-value coverage.

Prompt for coverage dimensions instead of raw quantity:

Generate tests covering:

- Happy path with valid input

- One boundary condition (empty, max, zero)  

- One error case that should raise/return error

- One state-dependent case (idempotent retry, cached result)



Each test must exercise a different code branch. Explain which branch in a comment.

We integrate this with Goatfied's compile-first workflow: after generation, we run coverage instrumentation and re-prompt if multiple tests hit identical line ranges. The model gets feedback like "Tests 2 and 4 both only cover lines 47–52; rewrite test 4 to trigger the else branch at line 58."

For languages with exhaustive pattern matching, pass the agent uncovered match arms from your coverage tool:

These branches are not covered by existing tests:



src/payment.rs:47 - PaymentMethod::Crypto(address) match arm

src/payment.rs:89 - Err(InvalidCurrency) branch  



Generate one test per uncovered branch. Each test must execute that branch and assert on its output.

Making branch divergence a requirement, not a suggestion, cuts manual test review time by half. The key is making coverage gaps machine-checkable and feeding them directly into the prompt.

Enumerate error paths before writing tests

Models default to happy-path coverage unless you explicitly demand otherwise. Generic "write error tests" prompts produce one or two token examples. The pattern that scales: walk the implementation's error paths first, enumerate failure modes, then generate one test per mode.

// Two-step prompt:

// 1. "Identify all error paths in this function (throw statements,

//    early returns with error payloads, validation failures). List them."

// 2. "For each error path, generate a test that triggers it with realistic

//    inputs production might see."



export function withdraw(account: Account, amount: number) {

  if (amount <= 0) throw new Error("Invalid amount");

  if (amount > account.balance) throw new Error("Insufficient funds");

  if (account.frozen) throw new Error("Account frozen");

  // ... happy path

}



// Expected test shape:

it("throws on negative withdrawal amount", () => { /* ... */ });

it("throws on amount exceeding balance", () => { /* ... */ });  

it("throws when account is frozen", () => { /* ... */ });

This mechanical enumeration prevents the common failure where your payment flow has no coverage for expired cards, rate limits, or transient network failures because the model assumed "error handling" meant one generic try-catch block.

Provide type witnesses to prevent schema drift

Generated tests often use simplified types or mock objects that don't match production data shapes. A test that passes with `{ id: 1, name: "test" }` may fail in production when that object carries 40 additional fields and nested references.

Provide type witnesses—real or anonymized production examples—and require tests to construct inputs from those templates:

// Provide this context in the prompt:

const productionOrderExample = {

  id: "ord_7x4k9",

  items: [{ sku: "WIDGET-A", qty: 2, taxable: true }],

  customer: { id: "cust_9m2", creditLimit: 5000 },

  shipping: { method: "express", address: { /* 15 fields */ } },

  metadata: { source: "web", campaign: null }

};



// Prompt: "Generate tests using objects shaped like productionOrderExample.

// Vary field values, but preserve structure."

Type witnesses surface incorrect assumptions: if your prompt includes an example and the model's generated test immediately strips half the fields, that's a signal the function signature or test setup is too permissive. This also handles the phantom dependency problem—tests that assume database state, network availability, or filesystem layout without setup. Pass your CI environment spec as context so the model knows exactly what's available:

Tests will run in a clean Docker container with:

- Empty /tmp directory

- No network access (stub HTTP with provided `mockClient`)

- Postgres 15 available at localhost:5432, empty schema



Any required files, DB records, or external state must be created in the test setup block. Show the setup code.

In Goatfied's managed runtime, this happens automatically—when generating tests for a repo with a defined devcontainer, the prompt includes the image name and exposed ports.

Decompose assertions for debuggability

Generated tests often produce single monolithic assertions: `expect(result).toEqual({ /* 50-line object */ })`. When this fails, you get a 200-line diff and no clue which field broke or why. Decompose assertions into granular checks, each testing one invariant:

// Instead of:

expect(processOrder(order)).toEqual({

  receiptId: "rec_abc123",

  total: 105.50,

  tax: 5.50,

  items: [{ sku: "WIDGET-A", price: 50 }]

});



// Prompt for:

const receipt = processOrder(order);

expect(receipt.receiptId).toMatch(/^rec_[a-z0-9]+$/);

expect(receipt.total).toBe(105.50);

expect(receipt.tax).toBe(5.50);

expect(receipt.items).toHaveLength(1);

expect(receipt.items[0].sku).toBe("WIDGET-A");

The prompt explicitly requests "separate assertion per property" and explains the debugging benefit. Decomposed assertions make CI logs actionable: instead of "expected object to equal object," you see "expected tax to be 5.50, got 6.00." In practice, this pattern also catches under-specified requirements—if the model struggles to decompose an assertion, the function likely returns an overly complex object.

For values that vary by run (timestamps, UUIDs, floats), explicitly constrain assertion style:

When asserting on non-deterministic values:

- Timestamps: check they're within a range (e.g., last 5 seconds)

- IDs: verify format (UUID shape) and uniqueness, not exact value

- Floats: use epsilon comparison (approx(expected, abs=0.01))



Show the matcher or helper used.

We pair this with post-generation linting. If we see `assertEquals(user.createdAt, '2025-01-15T...')` without a time-tolerance matcher, the plan step flags it for rewrite.

Enforce scope boundaries to prevent integration bloat

Models default to end-to-end style: spin up the server, hit the API, parse the JSON response. That's occasionally what you want, but if you asked for unit tests of a validation function, a 15-second test that boots Express is a failure.

Generate unit tests for `validateEmail(input: string)`. Tests must:

- Import only the function under test, not the full application

- Run in <50ms each

- Not require database, HTTP server, or file I/O



If the function has external dependencies, show how to mock them at the module boundary.

We enforce this in Goatfied's validation gate by running tests with a timeout. If a "unit test" takes more than 100ms, we re-prompt with the timeout violation and ask for isolated testing. The model usually figures out it needs to mock the DB client rather than start Postgres. The exception: if you're deliberately writing integration tests, say so up front. Models handle explicit intent better than trying to infer scope from vague keywords.

Incremental expansion over comprehensive generation

Generating an entire test suite in one shot is a recipe for unpredictable output. Better to grow the suite incrementally, validating each addition before moving to the next. We use a breadth-first strategy:

1. Generate happy-path tests first (simplest cases, fewest dependencies)

2. Validate and commit those

3. Generate edge cases and error conditions next, referring to the now-stable happy-path tests as context

4. Validate and commit

5. Generate integration or slow tests last, after unit coverage is solid

Each step is a small, reversible diff—Goatfied's core philosophy. If step 2 produces a flaky test, you roll back that single case without losing work from step 1. The model also gets better context: "Here are the working tests so far, now add error handling coverage."

More generation rounds means more latency, but the reduction in wasted retries makes this faster end-to-end. We've found 2–3 iterations hit the sweet spot: first pass gets structure right, second adds missing edge cases, third cleans up brittle assertions. Beyond that, you're usually fighting a deeper design problem in the code under test.

Validate tests actually catch regressions

Before merging generated tests, deliberately break the code being tested and verify the new tests catch it. If you can remove the filtering logic and the test still passes, the assertion isn't testing the right thing. This validation step catches tests that only exercise code paths without making meaningful assertions about correctness.

For teams generating tests at scale, this becomes a partial automation problem: run the new tests, apply a known breaking change to the source (flip a boolean, remove a condition), re-run, and confirm at least one test fails. If all tests still pass, surface that for human review before merge.

Handle partial success with explicit rollback

Not every generation attempt succeeds, and that's fine if your system accounts for it. The agent loop should treat partial success as a valid outcome. When a test batch fails validation, we:

• Keep any tests that passed lint + compile
• Mark failed tests for retry with annotated prompts ("previous attempt failed with: …")
• Limit retries (usually 2–3) before escalating to human review

This prevents the infinite loop problem ("model keeps regenerating the same broken test") and gives you a working subset to merge while a human fixes the hard cases. Goatfied's self-hosted option lets teams customize these thresholds—some prefer strict "all or nothing" validation, others are fine with 80% coverage and manual cleanup.

Composition and context windows

Large test suites hit context limits fast. You can't fit 50 existing tests plus module code into every prompt. We handle this with semantic chunking: embed existing tests, retrieve the 3–5 most similar to the target function, and use those as generation context. This keeps prompts focused and prevents the model from fixating on irrelevant examples. It also scales—a 10k-line test suite doesn't suddenly make generation impossible.

Build a feedback library from failures

The most underused operational asset: a structured log of what prompts produced broken tests and what constraints fixed them. When a generated test fails to compile or produces a false positive, capture the prompt, the failure output, and the corrected version. This library becomes your refinement dataset.

Over time, patterns emerge: maybe your codebase uses a custom test fixture pattern the model consistently misses, or your API returns `Result<T, E>` types that require specific unwrapping. Document these as prompt addenda: "Always use `setupTestFixture()` instead of manual mocks" or "API responses must be unwrapped with `.unwrap()` before assertions."

These aren't generic best practices—they're your codebase's specific idioms, learned from real failures. Feeding them back into generation requests compounds quality faster than tweaking models or frameworks.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)