Triaging flaky tests with AI assistance

AI tools help identify patterns in intermittent test failures by correlating logs, environmental factors, and code changes to surface root causes faster than manual triage.

A test fails on CI. You re-run it. It passes. The next day, a different developer hits the same failure on an unrelated branch. You've just lost an hour of collective time to a flaky test, and the real problem—whether it's a race condition, an environmental dependency, or a brittle assertion—remains undiagnosed.

Manual flaky test triage is expensive because the signal-to-noise ratio is terrible. Most failures look identical in logs. Developers waste time re-running builds or worse, merging code after seeing green on the third try. The underlying issues compound until teams either ignore test failures entirely or spend sprint time hunting intermittent bugs that should have been caught earlier.

AI assistance can compress the triage loop by surfacing patterns humans miss: correlating failures across branches, identifying environmental factors, and generating hypotheses about root causes before a human ever opens a log file. The key is structuring the problem so an AI agent has the right context and constraints to propose fixes without creating new brittleness.

Why flaky tests evade traditional tooling

Static analysis and standard test frameworks tell you what failed, but flaky tests fail inconsistently. A test might pass 19 times and fail once. Traditional CI tools can flag retry counts or failure rates, but they don't answer the diagnostic questions: Is this timing-dependent? Does it only fail on specific runners? Is the assertion itself non-deterministic?

The investigative work falls to engineers who manually diff logs, check runner configurations, and trace execution paths. This works when you have one or two flaky tests. It breaks down when you have dozens spread across a growing codebase, each failing unpredictably enough that no single developer owns the problem.

AI can help here because flaky test patterns are recognizable across large datasets. A test that fails only when run after TestX probably has shared state. A test that fails on slower CI runners likely has a hardcoded timeout. A test that fails more often on Mondays might depend on external services with maintenance windows. These patterns are tedious for humans to spot manually but straightforward for a model analyzing structured failure data.

Structuring test failure data for AI analysis

Effective AI triage starts with giving the model the right inputs. A single test failure in isolation is noisy. A hundred failures with metadata become a dataset you can reason about.

At minimum, you need:

Failure logs (stack traces, assertion diffs, stdout/stderr)
Test execution context (branch, commit SHA, runner OS/specs, retry attempt number)
Temporal data (timestamp, day of week, time since last green run)
Dependency snapshots (library versions, external service availability if you track it)

For example, if you're logging CI results to a structured store:


{

  "test_id": "tests/api/test_create_user.py::test_concurrent_creation",

  "status": "failed",

  "branch": "feature/user-endpoints",

  "commit": "a3f89b2",

  "runner": "ubuntu-22.04-4cpu",

  "attempt": 2,

  "timestamp": "2025-01-15T14:32:11Z",

  "error": "AssertionError: expected 2 users, found 3",

  "previous_failures": ["2025-01-14T09:12:03Z", "2025-01-10T16:45:22Z"]

}

This structure lets an AI agent ask: Does this test fail more on retries? Does it correlate with specific commits or dates? Is the error message consistent or does it vary?

Building an agent loop for triage

Goatfied's agent architecture—plan, constrain, edit, validate, retry—maps naturally to flaky test diagnosis. The agent doesn't just flag a test as flaky; it proposes a hypothesis, generates a fix, and validates the fix reduces failure rate.

Plan: The agent identifies a candidate flaky test (e.g., >10% failure rate with no recent code changes). It queries failure logs and test metadata to generate hypotheses: "This test may have a race condition in database transaction handling" or "This test appears to fail only on slower CI runners, suggesting a timeout issue."

Constrain: Before editing code, the agent checks what's in scope. Can it modify the test file? Can it adjust CI configuration? Are there lint rules prohibiting certain patterns (like bare time.sleep())? Constraints prevent the agent from introducing anti-patterns or touching production code when the issue is test-only.

Edit: The agent applies a small, targeted change. For a suspected race condition, it might add proper synchronization:


# Before

def test_concurrent_creation():

    thread1 = Thread(target=create_user, args=("alice",))

    thread2 = Thread(target=create_user, args=("bob",))

    thread1.start()

    thread2.start()

    # Assertion here without join



# After

def test_concurrent_creation():

    thread1 = Thread(target=create_user, args=("alice",))

    thread2 = Thread(target=create_user, args=("bob",))

    thread1.start()

    thread2.start()

    thread1.join(timeout=5)

    thread2.join(timeout=5)

    # Assertion now waits for threads to complete

For a timeout issue, it might replace hardcoded sleeps with polling:


# Before

time.sleep(2)  # Wait for async job

assert job.status == "completed"



# After

for _ in range(20):

    if job.status == "completed":

        break

    time.sleep(0.1)

else:

    raise TimeoutError("Job did not complete in 2s")

Validate: The agent doesn't merge the change immediately. It triggers a test run—ideally multiple runs—to confirm the failure rate drops. If the test still flakes, the agent retries with a different hypothesis.

Retry: If the first fix doesn't work, the agent logs the attempt and tries another approach (e.g., switching from synchronization to mocking an external dependency). The cycle continues until the test stabilizes or the agent escalates to a human with a summary of attempts.

When to let the agent auto-fix vs. escalate

Not every flaky test should be auto-fixed. The decision depends on risk and reversibility.

Auto-fix candidates:

Tests with clear patterns (consistent timeout errors, obvious race conditions)
Changes confined to test files (not production code)
Fixes that add safety without changing test intent (e.g., adding .join(), replacing hardcoded waits)

Escalate to human review:

Tests that fail in complex, non-reproducible ways
Suspected issues in production code (not just the test)
Fixes that would weaken assertions or hide real bugs

The agent should generate a pull request even for auto-fixes, but the validation gates differ. A low-risk fix (adding a timeout) might auto-merge after passing 10 consecutive runs. A higher-risk fix (changing an assertion) requires human approval.

Measuring triage effectiveness without inventing numbers

You'll know AI-assisted triage is working when you see:

Fewer manual re-runs: Developers stop reflexively hitting "Restart workflow" when a test fails.
Shorter PR cycle times: Less time blocked waiting to determine if a failure is relevant to the PR or environmental.
Clearer failure attribution: When a test fails, you have context (agent hypothesis, past attempts) instead of just a stack trace.

Track the ratio of flaky test failures to total test failures over time. If it's dropping, your triage process is working. If it's flat or rising, you may be auto-fixing symptoms without addressing root causes—a signal to tighten constraints or escalate more aggressively.

Integrating triage with PR workflows

Flaky test triage becomes more powerful when it's part of the PR review process, not a post-merge cleanup task. If an agent detects a flaky test during PR validation, it can:

Comment on the PR with the hypothesis and proposed fix
Open a linked PR with the fix for the test maintainer to review
Temporarily mark the test as flaky in CI config so it doesn't block unrelated work

This keeps the main PR moving while addressing the flake in parallel. It also surfaces test quality issues during code review, when context is fresh and the team is already engaged.

Staying compile-first and audit-friendly

AI triage should respect the same discipline as any automated code change: compile, lint, test, then propose. Goatfied's validation gates enforce this by default. An agent-generated fix must pass the same checks as a human-authored one before it's eligible for merge.

For teams with compliance requirements, every agent action should be logged: which test was flagged, what hypothesis was tested, what change was proposed, and whether it was auto-merged or escalated. This creates an audit trail showing not just what changed but why the agent believed the change was safe.