Pull Request Summaries That Engineers Actually Read Implementation: practical systems guide

A sprawling 47-file PR lands in your queue with a three-word description: "Fix auth issues." The diff touches authentication middleware, session handling, and half the API routes. Your team's security policy requires two reviewers before merge. You have twenty minutes before standup. This is when poorly written PR summaries turn security review into archaeology.

Good PR summaries don't just describe what changed—they provide the context, reasoning, and verification steps that let reviewers assess security implications quickly and confidently. Here's how to build a system that generates summaries engineers actually use for security review.

Why most automated PR summaries fail security review

Standard git commit messages and automated diff summaries optimize for describing mechanical changes: "Updated 12 files, added 340 lines." This works for tracking what happened but breaks down when reviewers need to assess security impact. The questions that matter—"Does this change authentication state management?" or "Can this introduce an authorization bypass?"—require understanding intent, not just syntax.

The disconnect widens with security-sensitive changes. A reviewer seeing `auth.verifyToken()` moved from middleware to a route handler needs to know: Was this deliberate? Does it maintain the same guarantees? What happens to requests that skip this route? Mechanical diffs don't answer these questions. Context-free AI summaries often hallucinate answers.

The three-layer summary structure

Effective PR summaries for security review need three distinct layers, each serving a different stage of the review process:

**Layer 1: The security-first abstract** (2-3 sentences). State what changed in the security model, not the codebase. "Moves rate limiting from nginx to application layer for per-user quotas. Maintains 429 responses but changes where they originate. No changes to authentication or authorization logic." This lets reviewers immediately triage severity.

**Layer 2: The architectural walkthrough** (3-5 bullets). Map major changes to system boundaries: which services/modules changed, what data flows shifted, which trust boundaries moved. Include what *didn't* change when that's surprising. "API gateway still validates JWTs before forwarding. Database queries now use prepared statements in all routes. Old string concatenation removed from admin panel only."

**Layer 3: The verification checklist** (4-8 concrete steps). This is where automation pays off. List the specific tests, lint checks, and manual verification you performed: "Ran `make test-auth` (18 test cases, all pass). Confirmed no eval/exec in git diff. Verified session tokens still expire after 1hr in local environment. Checked admin routes still return 403 for non-admin users."

Generating these with constrained automation

The agent loop in Goatfied—plan → constrain → edit → validate → retry—naturally produces these three layers if you constrain the generation correctly. The key is treating PR summary generation as a compilation target, not a creative writing task.

Start with constraints that force specificity:

constraints:

  - Must identify modified authentication or authorization code paths

  - Must list which security boundaries (network, process, data) changed

  - Must include test suite names that validate security properties

  - Cannot use phrases like "various", "several", "multiple" without specifics

  - Cannot claim tests passed without test command output

Then structure the prompt to extract this information from the actual diff and test output:

const summaryPrompt = `

Security-focused PR summary for: ${prTitle}



Files changed: ${changedFiles.join(', ')}

Test output: ${testResults}



Required sections:

1. Security model changes (what guarantees changed, not just what code)

2. Boundary analysis (which trust boundaries, data flows, or access controls changed)

3. Verification steps performed (exact commands run, with results)



Do not speculate on changes not visible in the diff.

`;

The validation step should reject summaries that don't ground claims in the actual changes. If the summary says "maintains authentication flow" but the diff shows changes to token validation, the validation fails and the agent retries with tighter constraints.

Integrating with your review workflow

PR summaries live in a comment or description field, but the system that generates them needs to pull from multiple sources. At minimum:

• The git diff itself (obvious, but often the *only* input to naive implementations)
• Test suite output from your CI system (not just pass/fail, but which specific security tests ran)
• Lint and static analysis warnings, particularly from security-focused linters
• Any triggered security policies from your pre-commit hooks or build gates

In Goatfied's managed environment, this data flows through the standard compile-first gates: lint runs first, tests second, and summary generation happens after both complete. The summary agent sees failures and includes them: "Rate limiting tests failed on user endpoint (test_rate_limit_user). Change does not merge until fixed."

For self-hosted setups, you can trigger summary generation after CI completes via webhook:

# After CI completes

goatfied agent run \

  --task generate-pr-summary \

  --context diff=$(git diff main...HEAD) \

  --context tests=$(cat test-results.json) \

  --output pr-comment.md

What reviewers do with useful summaries

Watch what happens when summaries contain genuine signal. Reviewers start with the security abstract, decide triage priority (urgent review vs. routine), then read the architectural walkthrough to form a mental model before opening files. The verification checklist becomes an audit trail—did the author think about token expiration, did they test with invalid credentials, did they check for SQL injection?

This changes the review from "find all the problems" to "verify the author found the problems." It's faster and catches more because the author's context is still fresh. When your summary says "verified admin routes require authentication in test_admin_auth", the reviewer checks that test actually covers edge cases, not starting from scratch hunting for auth bugs.

The honest version: this only works if summaries are actually grounded. The moment a generated summary claims "all security tests pass" when they didn't run, or says "no authentication changes" when middleware moved, engineers stop reading them. Trustworthiness compounds; so does its absence.

Related reading

• [Security playbook #5: practical Goatfied tactics for shipping PRs](/blog/goatfied-security-playbook-5)
• [Security playbook #13: practical Goatfied tactics for shipping PRs](/blog/goatfied-security-playbook-13)