open-source
What we learned open-sourcing our agent runtime
Goatfied open-sourced their agent runtime and discovered that external users exposed architectural assumptions through unexpected bug reports, feature requests, and integration needs.
We released Goatfied's agent runtime under Apache 2.0 six months ago, expecting feedback on our plan-constrain-edit-validate loop. Instead, we got bug reports about Docker volume permissions, feature requests for custom validation hooks, and a pull request that reimplemented our retry logic to handle rate limits we'd never seen in production. Open-sourcing infrastructure you use internally is different from building a library people adopt—and the gap taught us more about our own architecture than two years of internal iteration.
The decision to open-source the runtime wasn't ideological. We needed organizations to run agents in their own environments without exposing proprietary code or credentials to our managed service. Self-hosting required a runtime they could audit, extend, and trust. What we didn't anticipate was how quickly external use cases would surface assumptions we'd baked into "flexible" abstractions.
The validation gate became everyone's integration point
Our agent loop relies on compile/lint/test gates between the edit and retry phases. Internally, we ran TypeScript compilation, ESLint with our shared config, and a small Jest suite. Open-sourcing meant people wanted Python linters, Rust's Clippy, Go's staticcheck—toolchains we'd never considered. The original ValidationGate interface looked like this:
interface ValidationGate {
validate(diff: Diff): Promise<ValidationResult>;
}
Simple enough. Except external users wanted to inject custom gates between compile and lint, skip certain gates based on file paths, and run gates in parallel when they were independent. Our internal usage had always been sequential and uniform. The interface didn't capture dependencies or concurrency constraints.
We ended up adding a GatePipeline abstraction that let users declare gates with explicit ordering and fan-out:
const pipeline = new GatePipeline()
.addGate('compile', compileGate)
.addParallelGates(['lint', 'format'], [lintGate, formatGate])
.addGate('test', testGate, { dependsOn: ['compile'] });
This wasn't overengineering—it reflected real workflows. One contributor ran Terraform validate and plan in parallel, then a custom policy check only if both passed. Another needed language-specific gates that only ran when the diff touched relevant file extensions. Our uniform internal pipeline had hidden the need for this flexibility.
Retries exposed our optimistic concurrency model
The retry phase reruns validation after the LLM attempts a fix. Internally, our agents worked on isolated feature branches with no concurrent editors. When a company deployed the runtime across a team, multiple agents started editing overlapping files. Our retry logic assumed the working tree state matched what the agent last validated.
The first production issue came from a team running parallel agents on a monorepo. Agent A would validate a TypeScript change, Agent B would update a shared type definition while A was waiting for the LLM, and A's retry would fail because the compilation context had shifted. Our retry implementation looked like:
async function retryWithFix(context: AgentContext, error: ValidationError) {
const fix = await llm.generateFix(error);
await applyDiff(context.workingDir, fix);
return validate(context); // assumed unchanged working tree
}
No git ref checks, no optimistic locks, no conflict detection. The fix required tracking a base commit SHA and rejecting retries if the working tree diverged:
async function retryWithFix(context: AgentContext, error: ValidationError) {
const currentSHA = await git.getHeadSHA(context.workingDir);
if (currentSHA !== context.baseSHA) {
throw new ConcurrencyError('Working tree changed during retry');
}
// ... rest of retry logic
}
We added a --lock-working-tree flag for teams that wanted exclusive access during the agent loop. It's slower but eliminates a class of retry failures. Internal usage never needed this because we controlled scheduling.
Docker isolation turned into a packaging problem
The runtime uses Docker containers to isolate validation steps. Internally, we mounted the working tree as a bind mount and ran containers with our standard base images. External users ran into permission mismatches (host UID 1000, container UID 0), missing dependencies (no Node.js in the Python base image), and network isolation issues (couldn't reach internal package registries).
Our original container setup was minimal:
const container = await docker.createContainer({
Image: 'goatfied/validator:latest',
Cmd: ['npx', 'tsc', '--noEmit'],
HostConfig: {
Binds: [`${workingDir}:/workspace`],
},
});
We ended up adding a plugin system for custom container configurations. Users could inject volume mounts, environment variables, and even swap the base image:
interface ContainerPlugin {
configureContainer(config: ContainerConfig): ContainerConfig;
}
class NpmRegistryPlugin implements ContainerPlugin {
configureContainer(config: ContainerConfig) {
return {
...config,
Env: [...(config.Env || []), `NPM_TOKEN=${process.env.NPM_TOKEN}`],
HostConfig: {
...config.HostConfig,
Binds: [...config.HostConfig.Binds, `${HOME}/.npmrc:/root/.npmrc:ro`],
},
};
}
}
This let teams solve their own packaging problems without us maintaining base images for every language and registry configuration. The tradeoff: our container setup code became more complex, and we had to document security implications of mounting host credentials.
Self-hosting revealed our managed service assumptions
The managed Goatfied platform handles secrets, scales agent workers, and persists logs to our observability stack. The open-source runtime assumed none of that existed. Early self-hosters asked: where do LLM API keys go? How do I scale beyond one agent? How do I debug a failed validation step when the container is gone?
We'd hardcoded assumptions about secret management (process.env.OPENAI_API_KEY), assumed a single-agent execution model, and cleaned up containers immediately after validation. The runtime worked great as a library but poorly as a standalone service.
We added:
- A secrets provider interface so teams could plug in Vault, AWS Secrets Manager, or Kubernetes secrets
- A work queue abstraction backed by Redis or SQS for multi-agent scheduling
- Configurable log persistence (stdout, file, or remote sink)
These weren't features we planned—they emerged from watching people try to operationalize the runtime without our managed infrastructure. The managed service still handles these concerns better (we run a tuned Temporal cluster for scheduling and structured logs go to our SIEM), but self-hosting needed basic answers.
What we'd do differently
Start with a working example that doesn't rely on your internal infrastructure. We should have built a standalone demo that ran the full agent loop with local validation and public LLM APIs before opening the repo. It would have surfaced the Docker, retry, and secrets issues immediately.
Document the happy path and the sharp edges. Our initial README showed the simplest case. We didn't explain that parallel agents need concurrency control, that custom gates need explicit ordering, or that Docker permission issues are common on Linux hosts. Early users hit these walls and assumed the runtime was broken.
Treat the open-source version as a separate product. We thought of it as "Goatfied without the managed service," but users saw it as infrastructure they'd integrate into existing workflows. That means different priorities: extensibility over convenience, pluggability over batteries-included defaults, and clear boundaries between the core loop and integration points.
Why we still recommend managed for most teams
The open-source runtime is production-ready, but operating it requires infrastructure expertise. You need to manage LLM API keys, scale workers, handle log retention, and debug validation failures when containers don't match your local environment. The managed service handles this and adds SOC 2 audit logs, enterprise SSO, and uptime SLAs.
Self-hosting makes sense when you have regulatory constraints, need air-gapped deployments, or want deep customization of the validation pipeline. For teams shipping features quickly and trusting external services for CI/CD, the managed option is faster. The open-source runtime proved we can support both without compromising the core agent loop.