Telemetry ethics for open-source assistants: a production guide

Building an AI coding assistant means solving a problem most vendors ignore: users want tools that learn from their patterns and improve over time, but they also need zero risk that proprietary code, client secrets, or internal architecture decisions leak outside their perimeter. Telemetry—the events, errors, and usage signals your assistant collects—sits at the heart of this tension. Design it poorly and you either ship a blind tool that can't improve, or you create a compliance nightmare that enterprise security teams block on sight.

This guide walks through the concrete systems decisions that make telemetry both useful and trustworthy. You'll see how to structure pipelines that respect user boundaries while still powering model improvements, debugging, and reliability measurement—and why the architectural choices you make on day one determine whether you can scale to regulated industries or remain stuck in hobbyist territory.

The baseline threat model

Define what "leakage" means before you write a single line of instrumentation code. For a coding assistant, three categories matter:

**Source code and literals**: function bodies, variable names, comments, hardcoded API keys, internal URLs. This is nuclear material—any transmission without explicit opt-in is unacceptable. A single leaked function signature can reveal unannounced products or architectural patterns a competitor would pay for.

**Structural metadata**: file paths, import graphs, language distribution, error types. Lower risk, but aggregate patterns still expose stack choices. If you log that `file_hash_A` imports `file_hash_B` and those files are in a public repo, a motivated attacker can brute-force the hashes and reconstruct your microservice topology.

**Behavioral signals**: which commands users run, how often edits fail validation, retry counts. Generally safe, but even "anonymous" usage patterns can expose workflow details when correlated with timestamps and public repositories.

Establish clear consent tiers from day one. Goatfied ships with three modes:

• **None**: Zero outbound telemetry. The assistant runs entirely local or in your VPC. We get nothing.
• **Aggregate**: Anonymized counters and error codes only—no file paths, no snippets. Sufficient for uptime dashboards and model A/B tests.
• **Full**: Structured event logs including sanitized diffs and validation traces, used to improve prompt engineering and catch edge cases.

Users pick at install time. Store the choice in a local config file, cryptographically signed so a compromised plugin can't silently escalate to "Full." Never bury this in a 40-page privacy policy—make it a first-run prompt that explains exactly what each tier collects and why.

Separate operational health from product analytics

Not all telemetry serves the same purpose. Split it into two streams with different infrastructures:

**Operational health**: error rates, latency percentiles, resource usage, validation gate pass/fail counts. These metrics keep your service running and help users debug their own deployments. Structure these as metrics (Prometheus, StatsD) that contain no user content and minimal context. A metric like `validation_failures{phase="compile",language="rust"} 42` tells you where the system breaks without revealing what anyone is building.

**Product analytics**: which features get used, where users abandon flows, what constraint rules fire most often. This informs roadmap decisions and is where you'll be tempted to over-collect. Strip all code content and file paths before these events hit the network.

For Goatfied's agent loop (plan → constrain → edit → validate → retry), operational telemetry tracks outcomes at each stage—compile success rate, lint rule violations by category, retry depth distributions—without ever seeing the code that triggered them. The agent loop already parses compiler errors to decide how to retry, so extracting structured error taxonomies adds minimal overhead.

A structured error event might look like:

{

  "event": "validation_failed",

  "stage": "compile",

  "category": "type_mismatch",

  "language": "typescript",

  "constructs": ["generic", "union"],

  "resolved_on_retry": true,

  "attempts": 2,

  "timestamp_utc": "2025-01-15T14:23:11Z"

}

This tells you "the model struggles with TypeScript generics in union contexts and usually fixes it by the second attempt" without touching any proprietary code. Product analytics stays in a separate database with a separate retention policy—operational metrics might live 90 days, product events get anonymized and aggregated immediately with raw data dropped after 7 days.

Design for explicit consent, not implicit acceptance

The standard software pattern—show a modal on first launch asking users to "help improve the product"—optimizes for opt-in rates rather than informed consent. This trains users to dismiss prompts without reading them and front-loads a trust decision before they've experienced any value from your tool.

Better: ship with telemetry *off* by default, then prompt for opt-in the first time a clear failure occurs—an edit that fails linting, a validation timeout, a retry loop that gives up. The prompt explains what data you'd collect (link to a public schema document) and frames it as "help us fix issues like this one." Users who just hit a pain point are more likely to consent because the value exchange is concrete and immediate.

For self-hosted Goatfied deployments, telemetry stays entirely within the customer's infrastructure unless they explicitly configure an egress endpoint. This shifts the trust boundary: enterprise teams can audit failures internally, then choose to share anonymized aggregates with upstream maintainers. That separation matters for compliance-heavy environments where even failure metadata can't leave the network perimeter.

Implementation details that matter:

• Store the consent decision locally, never server-side. Users audit telemetry state with `cat ~/.goatfied/config.toml`, not by logging into a web dashboard.
• Provide runtime inspection: a `goatfied telemetry status` command that shows the current setting and lists recent events queued for transmission.
• Support temporary opt-out: `GOATFIED_NO_TELEMETRY=1 goatfied edit` for one-off sessions on particularly sensitive code.
• Honor `HTTPS_PROXY` so users can route telemetry through inspection tools like `mitmproxy`.

Anonymization techniques that survive correlation attacks

Simply hashing identifiers or stripping names doesn't prevent re-identification when you're sending graph-structured data or time-series events. An attacker with access to your telemetry database and a guess about a specific user's activity can correlate events: "This session hit a TypeScript syntax error at 14:23, retried twice, then succeeded—that matches the timing of PRs in this public repo."

Practical defenses you can stack:

**Differential privacy budgets** for aggregate counts. Add calibrated noise so individual events can't be extracted. When Goatfied reports "Python edit success rate: 87% over 1,000 attempts," we add Laplace noise scaled to guarantee no single user's data swings the stat by more than 0.5%. This lets you track macro trends without exposing any one team's workflow.

**Jitter timestamps** by ±5 minutes before logging. This breaks fine-grained correlation while preserving hourly/daily failure trends. Rotate session IDs every 24 hours instead of tying them to machine identity—a persistent session ID across weeks becomes a tracking vector.

**Bucket rare error categories** into an "other" bin. If only three users hit a specific gRPC timeout variant, logging it verbatim makes them identifiable in your dataset. Similarly, truncate error strings to 200 characters and append `...` if longer—this preserves the error type while clipping away user-specific context that often appears at the end of stack traces.

**Sample aggressively** for high-volume events. You don't need every success ping; 1% sampling still surfaces statistically significant failure spikes and reduces your correlation attack surface.

For event-level logs (the "Full" tier), separate PII stripping from transmission. Each event passes through a local sanitizer that:

• Redacts string literals longer than 3 tokens
• Replaces file paths with language + depth (e.g., `python/3` instead of `src/services/auth/handler.py`)
• Strips comments entirely
• Hashes stable identifiers (user ID, repo ID) with a per-install salt

Only after that pipeline does the event hit the network. None of these techniques are perfect, but layered together they make correlation attacks expensive enough to deter casual snooping.

The validation telemetry dilemma

Compiler errors often contain snippets of the code that failed. A Rust borrow-checker error will quote the problematic lines. A TypeScript type mismatch will show the inferred vs. expected types. These are gold for improving the model's understanding of strict languages, but they're also precisely the literals you promised not to transmit.

The solution is a structured error taxonomy instead of forwarding raw compiler stderr. Parse errors into a schema:

{

  category: "type_mismatch",

  language: "typescript",

  constructs: ["generic", "union"],

  resolved_on_retry: true,

  attempts: 2

}

Goatfied's agent loop already parses errors to decide how to retry, so extracting this structure adds minimal overhead. You lose the exact line that failed, but you keep enough signal to know the model struggles with specific language features—and that insight directly improves constraint-phase prompts without touching any proprietary code.

For self-hosted users who want richer debugging, offer a **local-only error archive** that writes full stack traces to disk, encrypted with their own key. They can spot-check failures without sending anything outbound.

Use a two-pass filter before logging any error message that does get transmitted:

1. **Regex-based redaction** for common patterns—file paths, email addresses, API keys (even partial ones). Replace matches with `[REDACTED_PATH]`, `[REDACTED_EMAIL]`, etc.

2. **Length limits**: truncate error strings and clip away user-specific context.

For model-generated content, never log the raw output. Instead, log a hash of the output (useful for deduplication) and a boolean: "did this output pass validation?" If you need more debugging context, let users manually attach sanitized excerpts via a bug-report helper that shows them exactly what will be shared.

Make telemetry infrastructure auditable

Open sourcing your assistant isn't enough if the telemetry endpoint is a black box. Implement patterns that let security-conscious users verify your claims:

**Schema publication**: Maintain a versioned JSON schema for every telemetry event type in your documentation. Users can write their own analysis tools or third-party proxies that validate outbound events match published schemas. Treat this schema like a public API—when you add a field or change an event structure, bump a version number and document the change in release notes.

**Local logging with hash chains**: Write telemetry events to a SQLite ledger with a cryptographic hash chain (each event references the previous hash). This log never leaves the user's machine. If they've opted into "Aggregate" or "Full" mode, a separate process reads from the ledger, applies sanitization, and transmits. The ledger records which events were sent and when.

Users can inspect or export the ledger at any time to see exactly what crossed the boundary. This makes audits straightforward: point the compliance team at the ledger, show them the hashed chain proves no tampering, and let them diff the "sent" subset against your consent tier. No ambiguity about what went where.

**Dashboard for inspection**: The most trust-building feature is a local dashboard that shows users what your assistant has logged, with one-click export and purge. This doesn't have to be fancy—a JSON file in `~/.config/your-tool/telemetry.log` and a CLI command `your-tool telemetry show` will do. Provide a `--telemetry-dry-run` flag that prints events to stdout instead of shipping them. When a user can run `goatfied --telemetry-dry-run edit my_file.py` and see exactly what JSON blobs would leave their machine, you've eliminated the guesswork.

**Validation pipeline**: Run automated checks in CI that scan outbound telemetry for patterns that shouldn't exist—regex for AWS keys, file path structures, code snippets in events marked as "aggregate only." A failed check blocks the deploy, same as a failed test. This proves what you *didn't* capture, not just what you claim in docs.

The self-hosted escape hatch

The cleanest way to resolve telemetry ethics is to not transmit at all. Self-hosted deployments should run the full agent loop—planning, constraint checking, validation, retry—in the customer's own infrastructure with all telemetry staying in their Kubernetes cluster or VPC.

The key architectural decision: telemetry is a plugin boundary, not hardcoded instrumentation. The agent loop emits structured events to a local interface; what consumes those events is configurable. The managed cloud runs the full telemetry stack. Self-hosted deployments can point at `/dev/null`, a local Prometheus exporter, or the customer's own SIEM.

This matters for teams in regulated industries (healthcare, defense, finance) where even anonymized data export triggers a multi-month legal review. The tradeoff is they don't benefit from continuous model improvements trained on aggregated patterns, but for many users that's a feature, not a bug. Goatfied's self-hosted option gets the same compile-first reliability as the managed service without any data leaving their perimeter.

Emit structured events, not free-form logs. An event like `{type: "validation_failed", phase: "lint", duration_ms: 340, retry_eligible: true}` works in any backend. A log line "Linting failed after 340ms, will retry" requires parsing, breaks when you change wording, and is useless for aggregation. High-cardinality events (per-invocation timing) stay local, aggregate to metrics (p50/p99 latency) before export. Low-frequency, high-value events (user accepted a suggestion, validation passed on first try) go to your warehouse but shouldn't exceed a few MB/day even at scale.

Handle edge cases without dark patterns

Real-world telemetry systems encounter unusual states: network timeouts, disk full errors, users who toggle consent mid-session. How you handle these reveals your actual ethics.

**Don't queue indefinitely**: If telemetry transmission fails, retry with exponential backoff but cap the queue at 1,000 events. After that, drop the oldest events rather than filling disk. Users with network policies that block telemetry shouldn't end up with gigabytes of queued data.

**Allow selective opt-in**: Some users want to send crash reports but not usage analytics. Support granular categories rather than all-or-nothing consent. The three-tier system (None/Aggregate/Full) is a starting point, but consider splitting "Full" into separate research corpus consent.

**Clear data on opt-out**: When users flip `telemetry: enabled` to `false`, immediately purge the local queue. Don't try to "flush remaining events" first—that's exactly the behavior that erodes trust. Make deletion synchronous—when a user revokes consent, their data disappears in seconds, not "within 30 days per our DPA."

**Build a kill switch**: Every telemetry client should check a boolean flag on startup and a cached remote config at intervals. If that flag flips to "disable," all telemetry stops until the next explicit opt-in. No grace period, no queued events, no "we'll flush what's in memory first." This is your escape hatch when you discover you've been collecting something you shouldn't.

**Document retention**: Be specific about server-side retention. Goatfied keeps aggregate metrics indefinitely but individual events for 90 days before deletion. Users who request data export get JSON dumps of events tied to their installation UUID. Make this query available via API so users can audit their own data without filing support tickets.

Publish aggregated insights, not raw data

If you claim to use telemetry to improve the product, show your work. Publish quarterly summaries: "42% of compile failures in the agent loop were due to missing imports; we shipped an improved import suggester." Give percentages, trends, and anonymized category breakdowns on a public dashboard.

This serves two purposes: it proves the data is actually useful, and it demonstrates you're aggregating and anonymizing properly. If you can't publish insights without leaking sensitive details, your collection is probably too granular. Transparency turns telemetry from a black box into a community resource—users can see whether their environment is an outlier, and contributors can prioritize fixes based on real impact.

The hardest part isn't the initial implementation—it's maintaining discipline as your product evolves. Resist the pressure to "just add this one extra field" when debugging production issues. Every expansion of telemetry scope should trigger the same review rigor as the initial design. When Goatfied added support for validating generated code against integration tests, we logged pass/fail outcomes but not which APIs the tests exercised. That made debugging harder, but it kept us aligned with our architectural constraint: instrument outcomes, not inputs.

Developers are better than most users at detecting mismatches between promises and behavior. Shipping an open source assistant with ethical telemetry isn't charity—it's recognizing that your users have the technical skills to verify your claims and the leverage to fork your project if you lie.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)