Prompt Patterns For Reliable Test Generation Design Tradeoffs: practical systems guide

Generating reliable tests from prompts is where most teams discover that "just ask the LLM" strategies fall apart. A passing test suite one run, brittle failures the next, and no clear signal whether the problem is the prompt, the code, or the model's mood. After a year of building Goatfied's test-generation agent loop—plan, constrain, edit, validate, retry—we've learned that prompt design is a systems problem, not a writing problem.

The real challenge isn't getting an LLM to produce syntactically valid test code. It's making generation **repeatable under constraint**, **debuggable when it drifts**, and **correctable without human rewrites**. This guide walks through the architectural patterns and tradeoffs we use in production, grounded in what actually survives contact with large codebases and non-deterministic models.

Why test generation is harder than it looks

Most generative coding workflows treat the LLM as a one-shot compiler: feed in context, get back code, hope it works. Test generation multiplies the failure surface. You're asking the model to infer:

• **What** to test (coverage intent)
• **How** to set up fixtures and state
• **Which** assertions express correctness
• **When** mocks vs. integration boundaries make sense

Each of these is a judgment call with context dependencies—framework conventions, project style, existing test patterns. A prompt that works beautifully for a simple TypeScript utility will produce garbage for a database-backed service with lifecycle hooks.

The brittle path is to cram all of this into a single mega-prompt and pray. The durable path is to **decompose the problem into constrained, validatable steps** that survive partial failures.

Pattern 1: Separate intent from implementation

The most useful shift we made was splitting test generation into two phases: **coverage planning** and **test implementation**. The planning step produces a structured artifact—not code, but a test spec:

{

  "target": "src/pipeline/executor.ts",

  "cases": [

    {"name": "retries_on_transient_failure", "type": "integration"},

    {"name": "respects_max_retry_limit", "type": "unit"}

  ]

}

This lets you:

• **Validate** coverage intent before writing a line of test code (does this match the team's testing strategy?)
• **Regenerate** implementations independently if the first attempt fails lint or runtime checks
• **Diff** plans across runs to see if the model is hallucinating new test cases

The planning prompt is narrow: "Given this module and these existing tests, list missing coverage areas as a JSON array." Implementation prompts then work case-by-case: "Write a test for `retries_on_transient_failure` that validates exponential backoff."

Tradeoff: Two LLM calls instead of one. In practice, planning is fast (low token count) and catches conceptual mistakes early, which saves expensive retries later.

Pattern 2: Constraint-first prompts beat example-heavy prompts

Developers instinctively reach for few-shot examples—"here's a good test, write one like this." It works until the model overfits to the example's structure and produces tests that look right but assert nothing meaningful.

We get better results from **explicit constraints as negative requirements**:

> Write a test for `parseConfig` that:

> - Uses the existing `mockFs` fixture (do not create a new filesystem mock)

> - Asserts on the returned object shape, not implementation details

> - Does NOT test error messages verbatim (check error type only)

This forces the model to solve within boundaries rather than template-match. When generation fails validation (say, it invents a new mock), the retry prompt inherits the constraint and the error message, which gives the model concrete correction signal.

Goatfied's validation gates—compile, lint, type-check—feed directly into the retry loop. A test that fails `tsc` gets re-prompted with the exact compiler error. A test that fails `eslint` gets the rule violation. This turns the prompt into a **constraint solver with feedback**, not a creative writer.

Pattern 3: Anchor to existing test infrastructure

The fastest way to generate unusable tests is to let the LLM invent setup code from scratch. Every project has idioms—specific test runners, custom matchers, fixture factories. Ignoring these means tests that don't run or require manual surgery to integrate.

We solve this with **infrastructure extraction**:

1. Parse the existing test directory to find common patterns (imports, setup hooks, helper functions)

2. Inject a schema of "available fixtures" into the generation context

3. Constrain the prompt: "Use `createTestUser()` from `fixtures/auth`, do not write your own user factory"

This reduces the model's creative surface area and ties generated code to the project's actual test harness. When we added this, our "test passes on first run" rate jumped from ~40% to ~75% for typical CRUD modules.

Example extraction might look like:

// Extracted from test/helpers.ts

const availableHelpers = [

  "setupTestDb(): Promise<DbConnection>",

  "teardownTestDb(conn: DbConnection): Promise<void>",

  "mockApiClient(responses: object): ApiClient"

];

Then the prompt references this list directly: "Select appropriate helpers from: …"

Pattern 4: Incremental expansion over comprehensive generation

Generating an entire test suite in one shot is a recipe for unpredictable output. Better to grow the suite incrementally, validating each addition before moving to the next.

We use a **breadth-first strategy**:

1. Generate happy-path tests first (simplest cases, fewest dependencies)

2. Validate and commit those

3. Generate edge cases and error conditions next, referring to the now-stable happy-path tests as context

4. Validate and commit

5. Generate integration or slow tests last, after unit coverage is solid

Each step is a small, reversible diff—Goatfied's core philosophy. If step 2 produces a flaky test, you roll back that single case without losing the work from step 1. The model also gets better context: "Here are the working tests so far, now add error handling coverage."

Tradeoff: More generation rounds means more latency. In practice, the reduction in wasted retries makes this faster end-to-end.

Pattern 5: Explicit rollback and partial success

Not every generation attempt succeeds, and that's fine if your system accounts for it. The agent loop should treat partial success as a valid outcome.

When a test batch fails validation, we:

• Keep any tests that passed lint + compile
• Mark failed tests for retry with annotated prompts ("previous attempt failed with: …")
• Limit retries (usually 2-3) before escalating to human review

This prevents the infinite loop problem ("model keeps regenerating the same broken test") and gives you a working subset to merge while a human fixes the hard cases.

Goatfied's self-hosted option lets teams customize these thresholds—some prefer strict "all or nothing" validation, others are fine with 80% coverage and manual cleanup.

Composition and context windows

Large test suites hit context limits fast. You can't fit 50 existing tests plus module code into every prompt. We handle this with **semantic chunking**: embed existing tests, retrieve the 3-5 most similar to the target function, and use those as generation context.

This keeps prompts focused and prevents the model from fixating on irrelevant examples. It also scales: a 10k-line test suite doesn't suddenly make generation impossible.

Related reading

• [Why we open-sourced Goatfied (and what we kept proprietary)](/blog/why-we-open-sourced-goatfied-and-what-we-kept-proprietary)
• [Open-Source playbook #3: practical Goatfied tactics for shipping PRs](/blog/goatfied-open-source-playbook-3)