Secret detection before agent execution: a production playbook

The moment an LLM agent writes `AWS_SECRET_KEY=AKIAXYZ...` into a file and your CI pipeline pushes it to a public repo, you've got seconds before automated scrapers find it. A credential scanner that runs after your agent merges to main is a compliance checkbox; one that runs before execution starts is a kill switch. The question isn't whether to detect secrets—it's where in the agent loop you enforce the gate, and whether that gate actually stops leaks or just documents them after the fact.

The core tension: you need detection tight enough to catch real leaks, fast enough that developers don't route around it, and specific enough that false positives don't erode trust. Get the balance wrong and you either leak credentials or train engineers to ignore alerts.

Why agents leak secrets differently than humans

Agents generate secret leaks through patterns humans rarely exhibit. An LLM might inline credentials during a refactor, copy-paste from example docs that include demo keys, or populate environment variables in test fixtures because the training data showed similar examples. The agent has no concept of "I know this looks like a key but it's actually a placeholder."

Common agent-specific patterns you won't find in human code:

**Overfit test fixtures**: An agent generates a mock OAuth flow and fills `client_secret=` with a real credential from the environment. The test passes because the agent never calls the auth endpoint. The secret sits in `fixtures/auth_test.go` until someone runs a scanner.

**Logging middleware that dumps headers**: Agents love adding observability. A new HTTP handler logs `request.Headers` verbatim, including `Authorization: Bearer sk_live_...`. This leaks customer tokens into Datadog or CloudWatch.

**Inline documentation with live examples**: An agent writes a README and copy-pastes a curl command from its own execution context—the key is real, the README goes into the public repo.

**Environment variable fallbacks**: Code like `api_key = os.getenv("GOATFIED_API_KEY", "ak_test_hardcoded_fallback")` where the default might be a key the agent saw in training data or pulled from an earlier execution step.

When agents iterate quickly—generating three variations of a config file in rapid succession—a single missed secret propagates across multiple retries before a human reviews the PR.

Pre-execution vs. pre-commit vs. post-commit scanning

Most implementations scan at one of three points, each with different failure modes:

**Post-commit scanning** during CI catches everything but at the worst possible time. The agent has already burned tokens iterating on broken code, the secret has touched multiple systems—your repo, build logs, container registries. By the time a post-commit hook fires, the agent has moved on. Reverting requires manual intervention, and the credential is already in Git history. This works as a safety net but shouldn't be your primary defense.

**Pre-commit scanning** (GitHub push protection, git hooks) tells you "we almost shipped this." But agents don't have the social contract humans do. An LLM editing files in a loop has no `--no-verify` escape hatch in its mental model. It will happily generate realistic-looking AWS access keys in example code, commit them, open a PR, and move on.

**Pre-execution scanning** catches secrets before the agent loop begins its plan-edit-validate cycle. The edit is rejected before `git add`, before the compiler sees it, before it lands in any audit log. In Goatfied's agent loop (plan → constrain → edit → validate → retry), this means running detection during the **constrain** phase—after the agent proposes a plan but before any file edits hit disk.

The operational difference is stark: pre-commit scanning creates remediation work, pre-execution scanning prevents the leak entirely. If a secret appears in the planned changeset, the loop halts and the agent retries with specific feedback.

Pattern engines: regex, entropy, and validated matches

Build detection as a tiered strategy. Each tier trades precision for coverage:

**Tier 1: High-confidence patterns** (regex for known formats). AWS keys start with `AKIA[0-9A-Z]{16}`, GitHub tokens follow `ghp_[a-zA-Z0-9]{36}`, Slack webhooks have recognizable prefixes. These rarely false-positive and should hard-block execution. Run fast regexes first—this gives you ~95% coverage of known secret formats with sub-100ms latency.

**Tier 2: Entropy-based scanning**. Any 32+ character alphanumeric string with high Shannon entropy (typically 4.5+ bits per character) deserves scrutiny. This catches generic API keys, base64-encoded credentials, and custom tokens that don't follow public patterns. The tradeoff: you'll flag base64-encoded test fixtures, UUIDs, and minified bundles. Use entropy as a secondary heuristic—apply it only to strings that look vaguely credential-like (assigned to variables named `key`, `token`, `password`, or in common config file locations).

**Tier 3: Validated matches**. When you detect an AWS key, make a lightweight API call to `sts:GetCallerIdentity` to confirm it's real. GitHub tokens can be validated against their `/user` endpoint. This eliminates false positives from example code and placeholder values, but adds 100-500ms latency and requires careful rate limit handling. Reserve this for critical findings before blocking a deployment.

Context matters more than most documentation admits. The string `password=test123` in `docker-compose.yml` is probably real. The same string in `test/fixtures/invalid-login.json` is not. Your scanner needs to understand file paths and ideally AST position. Scanning raw text without context is how you end up flagging every example in your documentation.

Integrating detection into the Goatfied constraint phase

In Goatfied's agent loop, the **constrain** step happens after planning but before the agent writes any code. This is where you inject rules: "Don't modify files in `/vendor`," "Keep functions under 50 lines," "No content matching these secret patterns may appear in any proposed edit."

Before the agent writes a diff, run the proposed content through your pattern library. If a match is found, reject the edit immediately and return specific feedback: "Proposed change to `config.yml` contains a pattern matching an AWS access key (line 14). Use environment variable interpolation instead." The agent can then retry with a different approach—often by factoring the credential into an env var or a secrets manager reference.

def validate_no_secrets(proposed_diff: str) -> ValidationResult:

    findings = []

    

    # Tier 1: known patterns

    for pattern, label in HIGH_CONFIDENCE_PATTERNS:

        matches = pattern.findall(proposed_diff)

        findings.extend([{"type": label, "line": m.line, "severity": "critical"} 

                         for m in matches])

    

    # Tier 2: entropy on credential-like contexts

    for token in extract_credential_candidates(proposed_diff, min_length=32):

        if shannon_entropy(token) > 4.5:

            findings.append({"type": "high_entropy", "value": token, "severity": "warning"})

    

    if findings:

        return ValidationResult(

            valid=False,

            message=f"Line {findings[0]['line']}: matches {findings[0]['type']}. "

                    "Use environment variables or a secrets manager."

        )

    return ValidationResult(valid=True)

This runs in single-digit milliseconds for typical diffs. No external process, no filesystem I/O. Just regex against the string buffer. In a tight loop where the agent is making 20-30 file changes to debug a failing test, sub-100ms p99 scanning keeps the system responsive.

This turns detection into a teaching moment. The agent sees the failure, understands what "secret detected" means in context, and typically fixes it on the next iteration. In Goatfied's loop, this feedback happens at the constrain step—before the agent even attempts the edit. We parse the planned changes, run detection against the intent, and if we catch something, we modify the constraint and let the agent replan. This costs one extra LLM call but saves the filesystem churn of writing and reverting files.

Layering in full-diff validation before commit

Pre-execution detection in the constraint phase isn't a replacement for commit-time scanning; it's a first line of defense. Before the agent opens a PR, run a full-repository scan with something like `detect-secrets` or `gitleaks`. This catches anything that slipped through: secrets in files the agent didn't modify but pulled from a template, or high-entropy strings that only look suspicious in aggregate.

In Goatfied, this happens in the **validate** phase after all edits are complete but before the diff is committed. The agent has generated a changeset, the code compiles, tests pass, and now you scan the entire diff. If a secret is found here, the agent retries from the plan step with additional context: "Scan detected credential in `config/database.yml`. Refactor to use environment variables before proceeding."

# Install gitleaks for commit-time validation

gitleaks detect --source . --no-git --exit-code 1 --redact

In your `goatfied.yaml`:

validate:

  - compile

  - lint

  - test

  - run: gitleaks protect --staged --no-banner

    on_failure: retry_with_context

When gitleaks finds a match, the agent gets a message like:

Secret detected: AWS Access Key in src/deploy.sh:12

Remediation: Store credentials in environment variables and reference via process.env or equivalent.

This two-layer approach—fast patterns in the constrain loop, comprehensive scanning before commit—gives you speed without sacrificing coverage.

Handling false positives without drowning signal

Every production system eventually adds allow-lists, despite everyone's best intentions. Test fixtures need fake secrets. Documentation needs examples. Some legacy systems use hardcoded keys for localhost development that are genuinely safe.

The false positive budget is tighter than the latency budget. Three false positives in a session and developers start ignoring all warnings. This is where precision matters more than recall—better to miss an edge case than to cry wolf on legitimate code.

Use inline annotations for known safe values:

# goatfied:secret-safe

TEST_API_KEY = "sk_test_51HqJ..." 

Your scanner parses these markers and skips annotated lines. This requires human judgment—the annotation is an assertion that "I reviewed this and it's safe."

For structural allow-lists, define rules by path pattern rather than per-instance exceptions:

# .goatfied/secret-allowlist.yml

structural_rules:

  - path: "test/fixtures/**"

    reason: "Test data only, no real credentials"

  - path: "docs/examples/**"

    entropy_threshold: 5.5  # Higher bar for docs

    reason: "Public documentation placeholders"

This makes the intention explicit and ties it to architectural patterns. Better than per-file `.secretallow` annotations that add friction, or repo-level blanket exceptions that hide real leaks.

For specific known-safe strings, maintain SHA-256 hashes with documented reasons:

allow:

  - hash: "a3c8f9..."  # example key in docs/quickstart.md

    reason: "public documentation placeholder"

  - hash: "7d2e1b..."  # test fixture

    reason: "hardcoded test data, rotated monthly"

The scanner computes hashes of detected secrets and checks the allow-list before blocking. This makes exceptions auditable—every allowed secret requires a documented reason—and prevents accidental re-introduction if the string changes.

Secrets already in the tree

Pre-execution scanning only helps if you also scan the current state. An agent editing an existing file with a committed secret will re-commit that secret unless you catch it.

On every agent loop start, run a full workspace scan and surface any findings as warnings. This won't block the loop—the damage is already done—but prevents agents from propagating secrets into new files or PRs. For remediation, the agent can propose a follow-up PR that replaces the literal value with an environment variable reference and adds a `.env.example` entry.

This turns a security incident into a configuration improvement and establishes a clean baseline: "We know the starting state is clean, so anything new came from the agent."

Operational telemetry and audit trails

Monitor your rejection rate by category. If you're blocking more test files than production code, your context detection is broken. If the same patterns get caught repeatedly, inject those specific examples into your system prompts so agents learn faster.

Track scanner performance in production:

• **Detection latency**: how long does constraint-phase scanning add to loop startup? Target sub-500ms for small diffs.
• **False positive rate**: what percentage of blocked loops were actually safe? Aim for <5%.
• **Secret prevalence**: how many secrets appear in agent-generated plans vs. human-authored code?

For compliance frameworks (SOC 2, ISO 27001), you need prevention and evidence. Running secret detection in the agent loop covers prevention—leaks never reach the repo. For evidence, emit structured logs for every validation run:

# Self-hosted: pipe gitleaks output to audit logs

gitleaks protect --staged --report-format json --report-path audit/secrets-$(date +%s).json

Goatfied's managed cloud captures this automatically: timestamp, files scanned, secrets found (redacted), remediation actions taken. Auditors can verify that every agent-generated diff passed secret detection before commit.

Self-hosted vs. managed detection

If you're running Goatfied self-hosted, you own the secret scanning pipeline. Deploy gitleaks or trufflehog as a sidecar in your CI runner, add it to the validation step, and control the rules, allow-lists, and retry behavior. Both the constraint-phase check and the pre-commit validation run in your infrastructure.

In Goatfied's managed cloud, secret detection runs automatically in both the constrain and validate phases. We use a curated ruleset that balances false-positive rates with coverage of common leak patterns (AWS keys, GitHub tokens, Stripe keys, etc.). You can extend it with custom patterns via your `goatfied.yaml`.

Both options block commits before they touch Git history. The difference is whether you maintain the scanner config yourself.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)