Secret Detection Before Agent Execution Design Tradeoffs: practical systems guide

The moment an LLM agent writes `AWS_SECRET_KEY=AKIAXYZ...` into a file and your CI pipeline pushes it to a public repo, you've got seconds before automated scrapers find it. Most teams discover this the hard way—through a breach, a compliance audit, or a panicked Slack message from security. The question isn't whether to catch secrets before execution; it's how to design a system that actually works without turning every code change into a manual review bottleneck.

The core tension: you need detection tight enough to catch real leaks, fast enough that developers don't route around it, and specific enough that false positives don't erode trust in the system. Get the balance wrong and you either leak credentials or train engineers to ignore alerts.

Scanning before vs. during the agent loop

Most implementations scan at one of three points: before the agent sees the codebase, after the agent proposes changes but before execution, or post-execution during CI. Each has different failure modes.

**Pre-loop scanning** (scanning the entire codebase before the agent starts) catches existing secrets but creates a cold-start problem. If you block agent execution on a full-repo scan, you're adding 30-90 seconds to every task. Worse, this approach assumes secrets are static—it won't catch a newly-generated API key the agent creates mid-loop. The real value here is establishing a clean baseline: "We know the starting state is clean, so anything new came from the agent."

**Mid-loop scanning** (checking each proposed diff before writing to disk) is where most production systems focus. The agent plans a change, you scan the diff, then either allow the write or reject it with feedback. This is the Goatfied model: constrain before edit. The tradeoff is computational—you're adding 100-500ms per file write, and you need a scanner that understands context well enough to avoid flagging test fixtures or example code.

**Post-execution scanning** during CI catches everything but at the worst possible time. The agent has already burned tokens iterating on broken code, and now you're rejecting the entire PR. If your CI takes 5 minutes to run, the agent is essentially flying blind for that entire window. This works as a safety net but shouldn't be your primary defense.

Entropy vs. pattern matching: the detection engine decision

Most secret scanners use regular expressions to match known patterns—AWS keys start with `AKIA`, GitHub tokens follow `ghp_[a-zA-Z0-9]{36}`, etc. This is fast and has near-zero false negatives for known formats. The problem is coverage: every new service needs a new regex, and developer-generated secrets (database passwords, internal API keys) follow arbitrary formats.

Entropy-based detection looks for high-randomness strings and flags them as potential secrets. A 32-character alphanumeric string has enough entropy to be suspicious. This catches unknown formats but generates false positives on hashes, UUIDs, and base64-encoded data. You'll flag things like `image: sha256:a3f5e8c...` constantly.

The practical approach most teams land on: pattern matching as the primary filter with entropy as a secondary heuristic. Run fast regexes first, then apply entropy analysis only to strings that look vaguely credential-like (assigned to variables named `key`, `token`, `password`, etc., or in common config file locations). This gives you ~95% coverage with manageable false positive rates.

Context matters more than most documentation admits. The string `password=test123` in `docker-compose.yml` is probably real. The same string in `test/fixtures/invalid-login.json` is not. Your scanner needs to understand file paths and ideally AST position. Scanning raw text without context is how you end up flagging every example in your documentation.

Feedback loops and agent retry behavior

When you detect a secret and block the write, what happens next determines whether the system works in practice. A naive implementation just returns an error. The agent sees the failure, doesn't understand what "secret detected" means, and tries the same change with minor variations until it hits retry limits.

Better implementations inject specific feedback into the agent's prompt: "The proposed change to `config/database.yml` contains what appears to be a hardcoded password. Please use environment variable interpolation instead." This turns detection into a teaching moment. The agent learns the pattern and typically fixes it on the next iteration.

In Goatfied's agent loop, this feedback happens at the constrain step—before the agent even attempts the edit. We parse the planned changes, run detection against the intent, and if we catch something, we modify the constraint and let the agent replan. This costs one extra LLM call but saves the filesystem churn of writing and reverting files.

The retry budget matters here. If detection is noisy, the agent burns through retries fighting with false positives. You need confidence in your scanner's precision—anything above 5% false positive rate and you're teaching the agent to work around the system rather than with it.

Where allow-lists inevitably show up

Every production system eventually adds allow-lists, despite everyone's best intentions. Test fixtures need fake secrets. Documentation needs examples. Some legacy systems use hardcoded keys for localhost development that are genuinely safe.

The question is how much ceremony you require. Per-file `.secretallow` annotations are maintainable but add friction. Repo-level config files (`secrets-allow.yml`) scale better but make it easier to accidentally allow too much. The worst pattern is in-line comments like `# pragma: allow-secret` that developers cargo-cult without understanding.

We've found the best balance is structural allow-lists: "Any file in `test/fixtures/` can contain example secrets" plus "Any value in `docker-compose.yml` tagged with `# local-only`". This makes the intention explicit and ties it to architectural patterns rather than per-instance decisions.

The operational reality check

Secret detection adds latency to every agent interaction. In a tight loop where the agent is making 20-30 file changes to debug a failing test, that latency compounds. You need sub-100ms p99 scanning or the system feels sluggish. This usually means running the scanner in-process rather than calling out to an external service.

The false positive budget is even tighter than the latency budget. Three false positives in a session and developers start ignoring all warnings. This is where precision matters more than recall—better to miss an edge case than to cry wolf on legitimate code.

Monitor your rejection rate by category. If you're blocking more test files than production code, your context detection is broken. If the same patterns get caught repeatedly, inject those specific examples into your system prompts so agents learn faster.

Related reading

• [The Goatfied agent loop: how we ship code that actually compiles first try](/blog/goatfied-agent-loop-compiles-first-try)
• [Agent-Loop playbook #1: practical Goatfied tactics for shipping PRs](/blog/goatfied-agent-loop-playbook-1)