Pr Triage Automation With Risk Scoring Production Playbook: practical systems guide

# PR Triage Automation With Risk Scoring: Production Playbook

Engineering teams that ship dozens of PRs daily face a coordination problem: which changes demand immediate security review, which can merge after CI passes, and which need specialized eyes before hitting production? Manual triage burns hours and creates inconsistent standards. The solution isn't blanket approval workflows—it's risk-based automation that routes each PR through the right gates.

This playbook walks through building a practical risk-scoring system for PR triage, the kind that runs in production at teams handling 50+ daily merges without creating review bottlenecks.

Why risk scoring beats fixed approval chains

Traditional approval workflows apply the same gates to every PR: two reviewers, security team sign-off, QA validation. This works for ten PRs a week. At scale, it creates two failure modes:

**Uniform friction**: A typo fix in documentation waits behind the same approval queue as a database migration. Engineers game the system by batching unrelated changes or fragmenting work to avoid "high-risk" labels.

**Review fatigue**: Security engineers become PR bottlenecks, skimming changes they don't have context for while missing the genuinely risky ones buried in the queue.

Risk scoring replaces this with dynamic routing. A PR that touches authentication code, modifies IAM policies, or changes how secrets are handled gets flagged for security review. A config tweak that only affects staging bypasses those gates entirely. The key is automatic classification based on diff content, not manual labeling.

Building the scoring dimensions

Effective risk scores combine multiple signals. Here's a starter set that covers most production scenarios:

**Blast radius**: How many users or systems does this change affect? A PR modifying a shared authentication library scores higher than one adjusting a single team's internal tool. Measure this by analyzing import graphs and deployment topology. If the changed file is imported by 20+ modules, or if it deploys to customer-facing infrastructure, weight it accordingly.

**Attack surface changes**: Does the PR expose new endpoints, modify authorization checks, or handle user input differently? Parse the diff for patterns like new HTTP routes, changes to permission decorators, or SQL query construction. Even small diffs that touch these areas deserve security eyes.

**Operational risk**: Database migrations, infrastructure-as-code changes, and configuration updates for rate limiters or circuit breakers carry deployment risk independent of security concerns. Flag these for SRE review based on file paths and change patterns.

**Change velocity**: A PR with 2,000 lines changed across 30 files is harder to review thoroughly than a 50-line focused change. High line counts don't automatically mean high risk, but they correlate with review gaps. Use this as a modifier, not a primary signal.

Implementation: scoring engine basics

A scoring engine reads PR metadata and diff content, applies weighted rules, and outputs a numeric risk level plus recommended review paths. Here's the logical flow:

interface PRRiskScore {

  total: number;

  dimensions: {

    blastRadius: number;

    attackSurface: number;

    operationalRisk: number;

  };

  requiredReviews: string[]; // e.g., ["security", "sre"]

  autoMergeEligible: boolean;

}



function scorePR(pr: PullRequest): PRRiskScore {

  const diff = parseDiff(pr.diffUrl);

  const impactedModules = analyzeImportGraph(diff.changedFiles);

  

  let score = {

    blastRadius: impactedModules.length > 15 ? 40 : 10,

    attackSurface: 0,

    operationalRisk: 0,

  };

  

  // Check for auth/security patterns

  if (diff.matches(/passport|jwt|session|authorization/i)) {

    score.attackSurface += 50;

  }

  

  // Infrastructure changes

  if (diff.includesFiles(['*.tf', 'k8s/*.yaml'])) {

    score.operationalRisk += 30;

  }

  

  const total = Object.values(score).reduce((a, b) => a + b, 0);

  

  return {

    total,

    dimensions: score,

    requiredReviews: deriveReviewers(score),

    autoMergeEligible: total < 20 && pr.ciPassed,

  };

}

The weights matter less than consistency. Start with a basic ruleset, run it against a month of historical PRs, and tune thresholds so high-risk matches your team's intuition about which changes warranted extra scrutiny.

Routing: from score to action

Scores become useful when they trigger automated workflows. At minimum, implement three tiers:

**Low risk (0-25 points)**: Auto-merge after CI passes if the author has write access. Post the risk breakdown as a PR comment for transparency, but don't block. This handles dependency bumps, documentation, and isolated feature toggles.

**Medium risk (26-60 points)**: Require standard code review from a teammate plus specialized review if specific dimensions are elevated (e.g., operational risk > 20 needs SRE eyes). Tag the appropriate teams automatically.

**High risk (61+ points)**: Block auto-merge, require both code review and security/SRE sign-off depending on dimension breakdown. For changes with attack surface scores above 40, trigger a security checklist that reviewers must acknowledge: "Does this PR validate user input? Does it modify authentication logic? Are secrets handled correctly?"

Goatfied integration for constraint enforcement

Risk scoring identifies what needs review, but it doesn't prevent risky patterns from reaching the PR stage. Integrate compile-time and pre-commit validation to catch issues earlier.

In Goatfied's agent loop, the **constrain** phase enforces project-specific security rules before code generation. Define constraints like "never construct SQL queries via string concatenation" or "all new endpoints must include rate limiting middleware." When the agent plans edits, these constraints act as hard boundaries—the generated code won't compile or pass lints if it violates them.

Pair this with the **validate** phase that runs tests, security scanners, and policy checks before the agent considers a change complete. If a PR emerges from this pipeline, you've already eliminated common vulnerability classes. Your risk scoring then focuses on architectural decisions and change coordination, not catching basic mistakes.

Example constraint for secret handling:

# .goatfied/constraints.yaml

- name: no-hardcoded-secrets

  pattern: '(password|api_key|secret)\s*=\s*["\x27]'

  scope: all

  severity: error

  message: Use environment variables or secret management

Any agent-generated code violating this fails validation before reaching your repository.

Monitoring and calibration

Track false positives (low-risk scores that caused incidents) and false negatives (high-risk scores that were routine changes). After 50+ scored PRs, review the distribution:

• Are 80% of PRs landing in low risk? If not, thresholds are too conservative.
• Do high-risk PRs actually correlate with post-deployment issues or security findings? If not, adjust dimension weights.
• Are certain file paths consistently over- or under-scored? Add path-specific rules.

Export scoring data alongside deployment outcomes. Correlate risk scores with rollback rates, incident reports, and security findings to prove the system works or identify gaps.

Related reading

• [Security playbook #5: practical Goatfied tactics for shipping PRs](/blog/goatfied-security-playbook-5)
• [Security playbook #13: practical Goatfied tactics for shipping PRs](/blog/goatfied-security-playbook-13)