open-source
Prompt patterns for reliable test generation
Learn three prompt patterns that help AI generate reliable, maintainable test cases with better coverage and fewer brittle assertions for your codebase.
Writing reliable test cases with AI often feels like rolling dice—sometimes you get comprehensive coverage, sometimes you get brittle assertions that break on the first refactor. The difference comes down to how you structure your prompt. When generating tests for open-source projects or internal codebases, specific prompt patterns dramatically improve consistency, coverage and maintainability compared to generic "write tests for this function" instructions.
Three patterns stand out in practice: the contract-first pattern (specify expected behavior before implementation details), the mutation-driven pattern (generate tests that would catch specific bugs), and the coverage-gap pattern (target untested branches). Each serves a different stage of test creation, and combining them creates a safety net that catches regressions without drowning your team in false positives.
Why generic test prompts fail
Most developers start with a prompt like "write unit tests for this code" and paste a function. The LLM dutifully generates tests—but they often assert implementation details rather than contracts, miss edge cases, or use magic numbers without explanation.
The core problem: the model lacks context about what matters. Should it test error paths? Boundary conditions? Thread safety? Without explicit guidance, it optimizes for code that looks like tests, not tests that prevent bugs.
Concrete example: A generic prompt for a date parser might generate tests for valid ISO dates but miss timezone edge cases, leap seconds, or what happens when the input buffer is exactly 255 bytes. The generated tests pass, but the first production incident reveals the gaps.
Contract-first pattern: define behavior, then verify
Start by specifying the contract—what promises this code makes to its callers. The LLM then generates tests that verify those promises, not just exercise the code.
Pattern structure:
Generate tests for [function/class] with this contract:
- Inputs: [describe valid and invalid ranges]
- Outputs: [describe return values, side effects, exceptions]
- Guarantees: [invariants, idempotency, atomicity]
- Edge cases: [empty inputs, boundaries, concurrent access]
[paste code]
Example for a cache implementation:
- Inputs: any hashable key, serializable values up to 1MB
- Outputs: value on hit, null on miss, raises CacheFullException when capacity exceeded
- Guarantees: LRU eviction, thread-safe reads, writes invalidate stale entries
- Edge cases: key collision, cache cleared during read, serialization failure
This pattern generates tests like "verify LRU eviction after 1000 inserts" and "concurrent reads return consistent values" rather than "test that get() calls internal _lookup()."
Mutation-driven pattern: test what could break
Instead of asking "what does this code do," ask "what bugs would we miss if we changed line X?" This shifts focus from happy paths to regression prevention.
Pattern structure:
Generate tests that would fail if we made these mutations to [code]:
- Remove null check on line 42
- Change < to <= in loop condition
- Swap order of validation steps
- Remove error handling block
The tests should catch each mutation independently.
For open-source libraries, this pattern is especially valuable because you can study historical bugs in the issue tracker and generate tests that would have caught them. Browse the "bug" label, identify the code that was fixed, and prompt for tests that fail on the buggy version.
Tools like Goatfied's BUILD suite can automate this flow: cloud coding agents generate the mutated versions, run candidate tests, and open pull requests with only the tests that demonstrate coverage gaps.
Coverage-gap pattern: fill untested branches
Parse your existing coverage report and target specific uncovered branches. This works particularly well for mature codebases where you're adding tests incrementally.
Pattern structure:
Coverage report shows these branches untested in [file]:
- Line 128: error path when database connection fails
- Line 256: pagination when results exceed 10k items
- Line 389: fallback when external API returns 429
Generate focused tests for each branch. Include:
- Setup to trigger the condition
- Assertion on the specific behavior
- Cleanup to avoid test pollution
Practical workflow:
1. Run coverage tool (pytest-cov, Istanbul, SimpleCov)
2. Export untested branches to JSON
3. Feed 5-10 branches per prompt—too many leads to superficial tests
4. Review and refine, especially mocks and fixtures
5. Commit tests in small batches with descriptive messages
Combining patterns for end-to-end coverage
Real projects benefit from layering these patterns:
1. Contract-first for new features: define the public API contract, generate comprehensive tests before implementation
2. Mutation-driven during code review: identify fragile logic, generate tests that would catch regressions
3. Coverage-gap for legacy code: systematically fill holes in existing test suites
Example workflow for an open-source PR:
- Contributor proposes a new sorting algorithm
- Maintainer pastes the contract: "must be stable, handle duplicates, work with custom comparators"
- Generate contract tests
- Review code, identify mutation points: "what if we skip the stability check?"
- Generate mutation tests
- Merge, then run coverage-gap pass on related modules
Validation checklist for generated tests
Before committing AI-generated tests, verify:
- [ ] Tests assert outcomes, not implementation details
- [ ] Each test fails for exactly one reason (no conjunctions in assertions)
- [ ] Setup and teardown are explicit, not hidden in conftest/beforeEach
- [ ] Mock objects have realistic behavior, not just return values
- [ ] Flaky tests (time, randomness, external services) are isolated or stabilized
- [ ] Test names describe the scenario and expected outcome
Red flags: Tests that never fail, tests that duplicate each other with different data, assertions on internal state instead of public API, or any test over 50 lines without clear sections.
Frequently asked questions
How do I prevent tests from becoming outdated when the code changes?
Include the contract or invariant as a comment in both the code and test file. When you prompt for updates, paste both the changed code and the original contract. Ask the LLM: "Does the contract still hold? If not, update the contract and regenerate tests." This forces an explicit decision about whether behavior changed intentionally.
What's the right balance between generated and hand-written tests?
Use generation for coverage—boundary conditions, error paths, permutations of inputs. Hand-write tests for complex scenarios that require domain knowledge: financial edge cases, security invariants, or subtle timing bugs you've actually encountered. A typical ratio might be 70% generated for breadth, 30% manual for depth.
How do I handle tests that depend on external services or databases?
Extend the contract pattern to specify fixture requirements: "Uses PostgreSQL test container with schema from migrations/001.sql" or "Mocks Stripe API with fixture from test/fixtures/successful_charge.json." The LLM will generate appropriate setup code. For open-source projects, prefer in-memory fakes (like H2 for SQL tests) to keep contributor setup simple.
Can I use these patterns with existing test frameworks?
Yes—these are prompting strategies, not tools. They work with pytest, Jest, RSpec, JUnit, or any framework. The key is providing structure in the prompt. If your framework has conventions (like pytest fixtures or RSpec let blocks), include an example test in the prompt so the LLM matches your style.
How do I measure whether the generated tests are actually useful?
Introduce a known bug (comment out a validation, flip a condition) and verify the tests catch it. Track test flakiness and false-positive rates—if generated tests break frequently on valid changes, tighten your contract specifications. For engineering teams, useful tests prevent production incidents per hour invested; measure that ratio over sprints.