Pr Triage Automation With Risk Scoring Implementation Guide: practical systems guide

# PR Triage Automation With Risk Scoring Implementation Guide: Practical Systems Guide

Most engineering teams handle pull requests the same way emergency rooms handled patients in the 1950s: first-come, first-served, regardless of severity. A two-line config typo waits in the queue behind a 500-line authentication refactor, while a critical security patch sits unreviewed because the usual reviewer is on vacation. The cost shows up as delayed incidents, context-switching overhead, and reviewers burning out on low-signal noise.

Risk scoring turns PR review into a deliberate allocation problem. Instead of treating every change identically, you classify by blast radius, code complexity, and operational sensitivity—then route high-risk work to appropriate gatekeepers while batching or auto-merging the routine. This guide walks through building a practical scoring system that integrates with your existing CI/CD pipeline.

Understanding the risk dimensions that matter

Effective risk scoring starts by defining what "risk" means in your codebase. Three dimensions capture most of the signal:

**Blast radius** measures how many systems a change can affect. Modifying a shared authentication library carries more weight than tweaking a feature flag in a single microservice. Calculate this by analyzing import graphs and deployment boundaries. If a file is imported by 50 other modules versus 3, the multiplier is obvious.

**Code complexity** signals cognitive load and bug surface area. Cyclomatic complexity, function length, and nesting depth all correlate with defect probability. A PR that adds 200 lines across 15 files with deep conditional logic scores higher than 200 lines of declarative configuration.

**Operational sensitivity** captures domain-specific context: changes to billing logic, cryptographic functions, or rate-limiting code demand more scrutiny than UI copy updates. You encode this through path-based rules (`src/payments/**` gets a 3x multiplier) or by tagging files in your repository metadata.

Combine these into a weighted score. A simple formula:

risk_score = (blast_radius * 0.4) + (complexity * 0.3) + (sensitivity * 0.3)

Tune the weights based on your team's incident history. If most outages trace to deployment configuration rather than application logic, weight sensitivity higher.

Calculating blast radius from dependency graphs

Building a blast radius metric requires mapping your codebase's import graph. Most languages expose this through static analysis tools—`jdeps` for Java, `go mod graph` for Go, or AST parsers for JavaScript.

Start by generating a directed graph where nodes are files and edges are imports:

# Example with a Python project

grep -r "^from.*import\|^import" src/ | \

  awk '{print $2, $1}' > imports.txt

Then compute centrality scores. Betweenness centrality works well here: files that sit on many import paths act as choke points. If `auth.ts` has a betweenness score of 0.8 while `footer.tsx` scores 0.02, you know which changes propagate widely.

Cache this graph and update it on merge to main. When a PR touches a file, look up its centrality score—that's your blast radius input. For files not in the graph (new additions), inherit the score from their nearest neighbor or default to a conservative mid-range value.

Integrating complexity analysis in CI

Code complexity metrics belong in your CI pipeline, not as a post-merge audit. Tools like `sonarqube`, `lizard`, or language-specific linters already compute cyclomatic complexity per function.

Add a CI step that:

1. Runs complexity analysis on changed files only (not the entire codebase—too slow)

2. Aggregates function-level scores into a file-level metric

3. Outputs a JSON artifact with per-file complexity values

# GitHub Actions example

- name: Calculate complexity

  run: |

    lizard src/ --format json > complexity.json

    jq '.files[] | select(.filename | IN(${{ github.event.pull_request.changed_files }}))' \

       complexity.json > pr_complexity.json

If a PR introduces functions with complexity over 15, flag them separately—even low blast radius code becomes risky when it's hard to reason about. Combine this with diff size: a 10-line change with complexity 20 deserves more attention than a 100-line change with complexity 5.

Encoding domain sensitivity rules

Sensitivity scoring is the most manual dimension but also the highest signal. Start with a simple path-based ruleset in a config file:

sensitivity_rules:

  - path: "src/payments/**"

    score: 10

  - path: "src/auth/**"

    score: 9

  - path: "config/production/**"

    score: 8

  - path: "docs/**"

    score: 1

  - path: "tests/**"

    score: 2

As your system matures, add function-level annotations. If a file handles both sensitive and routine operations, annotate the critical functions:

// @sensitive: handles-pii

function encryptUserData(data: UserData) {

  // ...

}

Parse these during analysis and boost the score accordingly. This keeps the ruleset maintainable as codebases evolve—you're not constantly updating path patterns.

Routing decisions based on composite scores

Once you have a risk score per PR, automate routing decisions. Define thresholds that trigger different workflows:

• **Score < 3**: Auto-merge after tests pass (typo fixes, docs, low-complexity test updates)
• **Score 3–7**: Standard review by any team member
• **Score 7–9**: Require review from a senior engineer or domain expert
• **Score > 9**: Mandate two approvals, one from security/platform team

Use GitHub's CODEOWNERS or GitLab's approval rules to enforce this programmatically. For extra rigor, Goatfied's constraint system can encode these policies as compile-time gates: a PR touching `src/payments/` won't merge without a specific reviewer's approval, enforced at the platform level rather than relying on manual vigilance.

Auto-merge the low-risk tail aggressively. If 40% of your PRs are documentation updates, dependency bumps, or test additions that score under 3, removing them from review queues frees reviewers to focus on the work that matters.

Handling false positives and calibration

No scoring system is perfect. You'll see false positives (a massive test file refactor flagged as high-risk) and false negatives (a subtle race condition in "simple" code). Track both through incident post-mortems and PR feedback.

Add an override mechanism: let PR authors tag a change as `risk-override: low` with a required justification. Log these overrides and review them quarterly. If one path consistently gets downgraded, your sensitivity rules need tuning.

Calibration happens iteratively. After 100 merged PRs, plot score distribution against actual incidents or rollbacks. If most incidents came from PRs scored 4–6, your thresholds are misaligned—tighten the mid-range or adjust dimension weights.

Related reading

• [Security playbook #5: practical Goatfied tactics for shipping PRs](/blog/goatfied-security-playbook-5)
• [Security playbook #13: practical Goatfied tactics for shipping PRs](/blog/goatfied-security-playbook-13)