Secret Detection Before Agent Execution Failure Patterns And: practical systems guide

A team pushes an automated PR to production. The CI passes. Two hours later, the service starts throwing 401s because a rotated API key leaked into a logging statement. The agent that wrote the code never saw a secret scanner—it just concatenated strings until tests went green.

Secret leaks in agent-generated code are more common than you'd expect. Models don't "see" secrets the way humans do. They pattern-match on variable names, copy examples from training data, and occasionally emit real credentials when context windows overflow. The failure mode isn't malicious—it's probabilistic. And because agents iterate fast, they can commit secrets across dozens of files before a human spots the problem.

This guide walks through where secret detection fits in the agent loop, what breaks when you defer it to post-commit hooks, and how to build checks that catch leaks before code enters version control.

Why post-commit secret scanning misses agent-generated leaks

Traditional secret scanning runs after `git commit` or during CI. GitHub's push protection and GitGuardian intercept secrets that touch the remote. For human developers, this works—most people notice when their editor auto-completes `GITHUB_TOKEN=ghp_...` and can amend the commit.

Agents don't have this feedback loop. When Goatfied's agent loop runs `plan -> constrain -> edit -> validate -> retry`, each cycle produces a candidate diff. If validation only checks syntax and tests, a secret buried in a new config file or a debug log passes through. By the time a post-commit hook fires, the agent has moved on. Reverting the commit requires manual intervention, and the secret is already in the Git history.

The fix: treat secret detection as a **constraint**, not a post-hoc audit. Run it in the `validate` phase, before any diff gets written to disk. If a secret is found, the agent retries with instructions to remove it—just like it would for a type error or linter violation.

Common agent secret patterns you won't find in human code

Agents leak secrets in ways humans rarely do:

**1. Overfit test fixtures**

An agent generates a mock OAuth flow and fills `client_secret=` with a real credential from the environment, because the training data showed similar examples. The test passes because the agent never calls the auth endpoint. The secret sits in `fixtures/auth_test.go` until someone runs a scanner.

**2. Logging middleware that dumps headers**

Agents love adding observability. A new HTTP handler logs `request.Headers` verbatim, including `Authorization: Bearer sk_live_...`. This leaks customer tokens into Datadog or CloudWatch.

**3. Inline documentation with live examples**

An agent writes a README and copy-pastes a curl command from its own execution context:

curl -H "X-API-Key: ak_prod_abc123..." https://api.goatfied.ai/v1/agents

The key is real. The README goes into the public repo.

**4. Environment variable fallbacks**

Agents sometimes write code like:

api_key = os.getenv("GOATFIED_API_KEY", "ak_test_hardcoded_fallback")

If the environment variable isn't set during testing, the agent picks a default. That default might be a key it saw in training data or pulled from an earlier execution step.

Where to run detection in the agent loop

In Goatfied's agent loop, secret detection runs in the `validate` phase, right after compile/lint/test checks and before the diff is committed. The sequence looks like:

1. **Plan**: Agent decides to add an S3 integration.

2. **Constrain**: Agent knows it must use environment variables, not hardcoded keys.

3. **Edit**: Agent writes `boto3.client('s3', aws_access_key_id=...)` with a placeholder.

4. **Validate**: Secret scanner runs. Finds no leaks. Compiler checks types. Tests pass.

5. If validation fails, **Retry**: Agent gets feedback like "Secret detected in `aws_manager.py:42` - replace with environment variable."

This loop means agents self-correct. A leaked key becomes a validation error, just like a missing import.

Building a pre-commit secret scanner for agents

You want a scanner that runs in milliseconds, blocks on matches, and provides clear remediation instructions. Here's a minimal setup using `gitleaks` (open-source, regex-based):

# Install gitleaks

brew install gitleaks  # or download binary



# Run on staged files before commit

gitleaks protect --staged --verbose

Wire this into your agent validation step. In Goatfied, you'd add it to your `goatfied.yaml`:

validate:

  - compile

  - lint

  - test

  - run: gitleaks protect --staged --no-banner

    on_failure: retry_with_context

When gitleaks finds a match, the agent gets a message like:

Secret detected: AWS Access Key in src/deploy.sh:12

Remediation: Store credentials in environment variables and reference via process.env or equivalent.

The agent's next edit removes the hardcoded key and injects an environment variable lookup.

Handling false positives without breaking the loop

Regex-based scanners flag anything that looks like a secret: test fixtures, example keys, and placeholder strings. If your agent hits a false positive, it'll retry indefinitely.

Solutions:

• **Allowlist known test patterns**: Gitleaks supports `.gitleaksignore` with line-level exemptions.
• **Use entropy detection**: Tools like `trufflehog` score randomness. A test key like `sk_test_1234567890` has low entropy. A real key like `sk_live_3kX9mP...` scores high.
• **Contextual rules**: If a key appears in `fixtures/` or is assigned to a variable named `EXAMPLE_KEY`, treat it as non-blocking.

In Goatfied, you configure this as part of your constraint set. The agent learns that test files can contain low-entropy secrets, but anything in `src/` or `config/` triggers a hard fail.

Self-hosted vs. managed detection

If you're running Goatfied self-hosted, you own the secret scanning pipeline. Deploy gitleaks or trufflehog as a sidecar in your CI runner, and add it to the validation step. You control the rules, the allowlists, and the retry behavior.

In Goatfied's managed cloud, secret detection runs automatically in the validation phase. We use a curated ruleset that balances false-positive rates with coverage of common leak patterns (AWS keys, GitHub tokens, Stripe keys, etc.). You can extend it with custom patterns via your `goatfied.yaml`.

Both options block commits before they touch Git history. The difference is whether you maintain the scanner config yourself.

Making secret detection audit-friendly

Compliance frameworks (SOC 2, ISO 27001) care about two things: **prevention** and **evidence**. Running secret detection in the agent loop covers prevention—leaks never reach the repo. For evidence, you need logs.

Every validation run should emit:

• Timestamp
• Files scanned
• Secrets found (redacted)
• Remediation actions taken

Goatfied's audit logs capture this automatically. For self-hosted setups, pipe gitleaks output to your SIEM or append to a structured log file:

gitleaks protect --staged --report-format json --report-path audit/secrets-$(date +%s).json

Auditors can verify that every agent-generated diff passed secret detection before commit.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)