Observability for AI coding assistants in production

Learn how to instrument AI coding assistants with LLM-specific observability: inference traces, quality metrics, token tracking, and model behavior correlation.

Engineers who've shipped an AI coding assistant to production quickly discover that traditional application monitoring doesn't cut it. You can't debug why an LLM-powered agent silently degraded to producing broken code or started burning through token budgets with a standard APM dashboard. The problem isn't infrastructure health—it's that LLMs are probabilistic, context-sensitive, and fail in ways that escape conventional metrics.

Observability for AI assistants in production demands a different toolkit: capturing LLM inference traces, tracking quality beyond HTTP 200s, understanding token economics in real time, and correlating model behavior with actual code outcomes. This post walks through what to instrument, how to structure telemetry, and practical techniques to make your AI coding assistant debuggable when it counts.

What makes LLM observability different

Traditional observability assumes deterministic behavior. Request → Process → Response. A 500 error is obvious; a slow database query shows up in traces. LLM-based systems break this model in three ways:

Probabilistic outputs. The same prompt can yield correct code, syntactically invalid code, or semantically broken logic across identical requests. You can't rely on status codes alone—what matters is whether the generated diff compiles, passes tests, and does what the user intended.

Context windows as state. Unlike stateless APIs, every LLM request carries expensive context: codebase snippets, conversation history, schema definitions. A degraded response might be due to truncated context, not model failure. Observability must surface what context actually reached the model.

Token economics. Cost isn't just compute time—it's directly proportional to tokens processed. A runaway context window or retry loop can silently 10x your spend without triggering latency alerts. Monitoring token usage per request, per user session, and across model tiers becomes first-class operational data.

Instrumenting the agent loop

Goatfied's agent architecture—plan, constrain, edit, validate, retry—provides natural trace boundaries. Each step generates observable signals that traditional metrics miss.

Planning phase. Log the initial intent (user prompt), retrieved context (file paths, line ranges, dependency metadata), and any constraints applied (language-specific rules, test coverage requirements). Capture this as structured data:


{

  "phase": "plan",

  "intent": "add pagination to search endpoint",

  "context": {

    "files": ["src/api/search.ts", "src/types/pagination.ts"],

    "dependencies": ["express", "zod"],

    "lines_retrieved": 145

  },

  "constraints": ["maintain_backwards_compatibility", "require_tests"]

}

Edit phase. Track the generated diff size, number of hunks, and whether the LLM adhered to the plan. A diff that touches 20 files when the plan scoped to 2 is a red flag. Correlate the prompt tokens sent versus completion tokens received—a sharp imbalance often signals the model went off-track.

Validation phase. This is where observability meets outcomes. Log compile results, linter errors, test pass/fail counts, and validation duration. Critically, correlate validation failures back to the edit phase's LLM trace. If 40% of diffs fail type checking, you need prompt engineering changes or tighter constraints, not infrastructure tuning.


{

  "phase": "validate",

  "compile": { "status": "failed", "errors": 2 },

  "lint": { "warnings": 5 },

  "tests": { "passed": 12, "failed": 1 },

  "validation_ms": 8400,

  "parent_trace_id": "edit-abc123"

}

Retry logic. Count retry attempts per task, the reason for each retry (validation failure, context overflow, rate limit), and track convergence. An assistant that takes 4 retries to land a simple fix is either over-constrained or receiving poor feedback signals. Surface this in dashboards as retry rate per intent type.

Structuring telemetry for debugging

Logs alone become noise at scale. Effective LLM observability depends on structured traces that connect user intent to final outcome across multiple model calls.

Adopt distributed tracing semantics. Each user request spawns a root trace. The agent's plan/constrain/edit/validate/retry cycle creates child spans. Within edit, the actual LLM API call is a leaf span with attributes for model ID, prompt tokens, completion tokens, temperature, and latency. Use OpenTelemetry conventions—most LLM providers now support OTEL exporters.

Tag traces with quality signals. Beyond technical success (200 OK), annotate spans with semantic outcomes:

code.compiled: boolean
tests.passed_count: integer
diff.lines_changed: integer
user.accepted: boolean (if you instrument feedback)

This lets you query: "Show me all traces where the code compiled but tests failed, grouped by model version."

Capture prompt templates and final prompts separately. Store the template used (with placeholders) and the actual rendered prompt. When model behavior shifts, you can diff whether your templates changed or if the filled-in context data changed. Storing full prompts is expensive; sample them (e.g., 1% of traffic, or all failing requests).

Token budget monitoring in real time

Token usage is both a cost center and a quality signal. Runaway token consumption often indicates a deeper problem—context bloat, infinite retry loops, or poorly scoped tasks.

Per-request budgets. Set hard limits on input tokens per phase. If retrieving context for a "fix typo" intent pulls 50k tokens, your context retrieval is broken. Enforce budget checks before sending to the LLM:


const contextBudget = getTokenBudget(intent.type);

const retrieved = retrieveContext(intent);

if (countTokens(retrieved) > contextBudget) {

  logWarning({ intent, retrieved, budgetExceeded: true });

  retrieved = truncateContext(retrieved, contextBudget);

}

Track token efficiency. Calculate tokens per successful diff. If you spend 100k tokens to generate a 3-line change that fails validation, and another 120k tokens on the retry that succeeds, that's 220k tokens for a 3-line diff. Surface this ratio in dashboards to identify inefficient intent patterns.

Model-tier usage. If you route simple tasks to cheaper models and complex tasks to premium models, monitor whether the routing logic works. A spike in GPT-4 usage for "add comment" intents suggests broken classification. Export token metrics by model tier, intent type, and success rate.

Correlating outcomes to model behavior

The ultimate question: did the assistant produce usable code? This requires joining telemetry from the LLM layer to compile/test/runtime layers.

Build a outcomes pipeline. Emit events at key decision points:

User accepts diff → outcome.accepted
User rejects diff → outcome.rejected (with optional reason)
Diff merged to main → outcome.merged
Diff reverted → outcome.reverted

Join these events back to the originating trace ID. Now you can answer: "What percentage of diffs generated by claude-3.5 with >10k context tokens actually got merged?" This is far more valuable than model latency percentiles.

Detect silent degradation. Track rolling averages of validation pass rates, acceptance rates, and retry counts. Set alerts on week-over-week changes. A 10% drop in pass rate might stem from a model provider update, a change in your prompt templates, or shifts in the codebase's complexity. Observability won't tell you why—but it tells you when to investigate.

Operational playbooks from telemetry

Raw telemetry is useless without actionable insights. Define specific queries and thresholds that drive operational decisions:

Token spend runaway: Alert if any user session exceeds 500k tokens in 10 minutes. Usually indicates a retry loop gone wrong.
Validation failure spike: If >30% of diffs fail compile in a 1-hour window, page on-call. Likely a prompt regression or model issue.
Context truncation rate: If >5% of requests hit context window limits, re-evaluate context retrieval logic.
Model latency SLOs: P95 latency >15s for edit phase? Consider switching model tiers or optimizing prompts.

In self-hosted deployments, export these metrics to your existing observability stack (Prometheus, Grafana, Datadog). In Goatfied's managed offering, we expose these via the platform dashboard, but the primitives remain the same.

Privacy and retention considerations

LLM telemetry often contains source code, user prompts, and proprietary logic. Treat observability data with the same sensitivity as production data:

Redact secrets. Strip API keys, credentials, and PII from logged prompts and context. Use allowlists, not denylists—assume everything is sensitive unless proven otherwise.
Short retention windows. Keep full prompt/completion payloads for days, not months. Aggregate metrics can persist longer.
Air-gapped telemetry. For environments that prohibit external telemetry export, deploy observability infrastructure within the same network boundary. Open-source stacks like Jaeger + Prometheus work well here.

Goatfied's self-hosted mode keeps all telemetry on your infrastructure by default. No traces leave your network unless you explicitly configure external exporters.