Model Routing Between Fast And Smart Engines Design: practical systems guide

Every production AI system eventually hits the same trade-off: the fast model isn't smart enough for complex requests, and the smart model is too slow—or expensive—for simple ones. Teams waste weeks building routing logic, only to discover that a static threshold on token count doesn't capture the difference between "fix this typo" and "refactor this module to use dependency injection."

The real challenge isn't picking a heuristic. It's designing a system that routes requests accurately without introducing new failure modes, tracks when routing decisions go wrong, and stays maintainable as your model catalog evolves. This guide walks through the architecture decisions that matter, based on patterns that hold up under production load.

Why naive routing breaks down

Most first attempts route on surface signals: request length, presence of file paths, or keyword matches like "refactor" versus "format." These work until they don't. A 50-token request asking "why does this authentication flow fail for OAuth providers" needs GPT-4, while a 500-token request pasting in a config file for formatting can go to Claude Haiku.

The core problem is that routing is itself an inference task. You're predicting which model will satisfy the user's intent with acceptable latency and cost. Doing this well requires:

• **Intent classification** beyond keywords—is this exploratory analysis, a targeted fix, or a batch transformation?
• **Context awareness**—does the request reference prior conversation turns that established constraints?
• **Fallback handling**—when the fast model produces garbage, can you retry with the smart model without the user noticing?

A brittle router creates a worse experience than always using the slow model. Users start second-guessing which phrasing will trigger the "good" model, and support tickets spike around "the AI gave me nonsense."

Decision architecture: where routing fits in the agent loop

In Goatfied's agent loop—plan → constrain → edit → validate → retry—routing happens **before** the plan step, but it's informed by signals from the validation phase of prior requests. The key insight: routing is stateful, not request-scoped.

Here's the data flow:

1. **Pre-routing classifier**: Extract features from the raw request—token count, parse complexity (can you identify specific line ranges?), presence of validation constraints like "must pass type-check."

2. **Route selection**: Choose model tier (fast/smart) and context window size. Log the decision with a unique request ID.

3. **Agent execution**: The selected model runs through plan/edit/validate. If validation fails (linter errors, test failures), that's a signal the fast model was insufficient.

4. **Feedback loop**: Failed validations with fast models increment a "complexity score" for similar future requests. This score decays over time as models improve.

Concretely, if a request to "add error handling to this HTTP client" routes to Claude Haiku and produces code that fails your linter, the next similar request automatically routes to GPT-4. The decay means you'll re-test Haiku in a week, capturing model updates without manual intervention.

Practical classifiers that work

Skip the LLM-calls-LLM approach for routing—it doubles your latency and cost. Instead, build a lightweight classifier from structured features:

**Syntactic complexity**: Run a fast parser (tree-sitter is 1–2ms) and count nesting depth, number of function definitions, and cyclomatic complexity. Requests targeting deeply nested code with high complexity go to the smart model.

**Validation requirements**: If the user includes "must compile" or references CI checks, route to smart. These constraints are explicit in Goatfied's interface, so you're not guessing.

**Historical edit scope**: Track the median diff size for this user's past requests. If they typically make 5-line changes and this request says "restructure the module," route smart. If they're asking for a typo fix, route fast.

**File type heuristics**: Terraform/YAML/JSON changes with no logic modification? Fast model. TypeScript with generic inference or Rust with lifetime annotations? Smart model.

Train a simple decision tree or logistic regression on 500 labeled examples (fast-success, fast-fail-retry, smart-only). You'll hit 85%+ accuracy in a day. The classifier runs in <10ms, adds negligible latency, and you can introspect its decisions in logs.

Handling fallback without user-visible retries

The failure mode that kills trust: the fast model returns broken code, the user sees it, *then* you re-run with the smart model. Now the user knows your system guessed wrong.

Instead, use **speculative dual execution** for borderline requests. Route to the fast model but *also* queue the smart model in the background. If the fast model's output passes your validation gates (compiles, lints, passes inline tests), return it immediately and cancel the smart-model job. If validation fails, stream the smart model's output without acknowledging the failure.

This adds cost for borderline cases, but typical workloads show 70–80% of requests clearly belong to one tier, leaving only 20–30% as speculative. For those, you're paying 1.5× instead of 2×, and you've eliminated user-facing retries.

Goatfied's validation phase (compile/lint/test gates) makes this viable: you have a fast, deterministic signal whether the output is usable, unlike systems where "quality" is subjective.

Cost and latency trade-offs you can measure

Set budgets at the request level, not the model level. For each incoming request, assign a latency target (p95 < 3s for fast, < 10s for smart) and a cost cap. If routing to the smart model would exceed the cap and the user hasn't explicitly requested high quality, fail fast with a message: "This request needs more compute than your tier allows—upgrade or simplify the scope."

Track these metrics per route:

• **Cost per satisfied request**: Total model cost divided by requests that passed validation. A high-cost fast model with 40% retries is worse than a slightly pricier smart model with 95% success.
• **Latency percentiles by intent**: "Explain this code" requests tolerate 10s p95; "fix this lint error" requests don't. Route aggressively for intents with tight latency budgets.
• **Retry rate by classifier confidence**: If your classifier was 90% confident and still routed wrong, your features are missing signal.

For a team running 10,000 requests/week, switching 30% of those from GPT-4 to Haiku (with a 15% retry rate) typically cuts model costs by 50–60% while keeping p95 latency under 4 seconds. The retry overhead is offset by the 3–5× speed and cost difference between tiers.

Operational hygiene: logging and drift detection

Your routing system will degrade silently as models update, user behavior shifts, and edge cases accumulate. Build observability from day one:

• **Log every routing decision** with features and outcome (success/retry/fail). Store these in structured logs queryable by date, user, intent, and model.
• **Weekly confusion matrix**: Which classifier predictions led to retries? Retrain your decision tree monthly with fresh data.
• **Model drift alerts**: If Haiku's retry rate jumps from 15% to 30% over two weeks, either the model regressed or user requests changed. Investigate before costs spike.

Self-hosted Goatfied deployments make this straightforward: all logs stay in your infrastructure, and you control the feedback loop. Managed deployments expose routing decisions via webhooks, so you can pipe them into your existing observability stack.

Related reading

• [Self-hosted AI coding assistants: a complete Goatfied deployment guide](/blog/self-hosted-ai-coding-assistants-goatfied-deployment-guide)
• [Deployment playbook #2: practical Goatfied tactics for shipping PRs](/blog/goatfied-deployment-playbook-2)