Model routing between fast and smart engines

Every production AI system eventually hits the same trade-off: the fast model isn't smart enough for complex requests, and the smart model is too slow—or expensive—for simple ones. The difference between a $0.002/1k-token call and a $0.06/1k-token call compounds fast at scale. Smart routing—sending simple queries to fast models and hard problems to capable ones—becomes a cost lever worth engineering properly.

The real challenge isn't picking a heuristic. It's designing a system that routes requests accurately without introducing new failure modes, tracks when routing decisions go wrong, and stays maintainable as your model catalog evolves. This guide walks through the architecture decisions that matter, based on patterns that hold up under production load.

Why naive routing breaks down

Most first attempts route on surface signals: request length, presence of file paths, or keyword matches like "refactor" versus "format." These work until they don't. A 50-token request asking "why does this authentication flow fail for OAuth providers" needs a frontier model, while a 500-token request pasting in a config file for formatting can go to GPT-4o-mini.

The core problem is that routing is itself an inference task. You're predicting which model will satisfy the user's intent with acceptable latency and cost. Doing this well requires:

• **Intent classification** beyond keywords—is this exploratory analysis, a targeted fix, or a batch transformation?
• **Context awareness**—does the request reference prior conversation turns that established constraints?
• **Fallback handling**—when the fast model produces garbage, can you retry with the smart model without the user noticing?

A brittle router creates a worse experience than always using the slow model. Users start second-guessing which phrasing will trigger the "good" model, and support tickets spike around "the AI gave me nonsense." Worse, some failure modes are silent: the fast model returns plausible-looking code that fails validation, costs accumulate from retries, and nobody notices until the monthly bill arrives.

The three-tier model strategy

Start with three tiers, not two. Most teams try fast/smart and discover they need a middle ground within a week.

**Tier 1 (sub-200ms)**: Small models for mechanical edits. Renaming variables, fixing linting errors, applying formatting rules, generating boilerplate from templates. These are the requests where any frontier model is overkill. If you're using Goatfied's agent loop, these are the tasks that hit the constraint phase and can be validated immediately—perfect candidates for a model that costs 100x less.

**Tier 2 (2-5 seconds)**: Mid-size models like GPT-4o-mini or Claude Haiku for standard feature work. Implementing a well-specified function, writing tests for existing code, refactoring a class that doesn't touch architectural boundaries. These succeed 85%+ of the time if your prompt includes sufficient context.

**Tier 3 (10-30 seconds)**: Frontier models like GPT-4, Claude Opus, or o1 for ambiguous or cross-cutting work. Resolving architectural tradeoffs, debugging race conditions, designing new abstractions. Save these for requests where the cost of getting it wrong exceeds the inference cost by orders of magnitude.

The mistake is trying to be clever about which Tier 2 model to use. Pick one and stick with it for a month. Your routing logic needs production data to improve, and A/B testing models simultaneously destroys your ability to learn.

Decision architecture: where routing fits in the agent loop

In Goatfied's agent loop—plan → constrain → edit → validate → retry—routing happens **before** the edit step, but it's informed by signals from the validation phase of prior requests. The key insight: routing is stateful, not request-scoped.

Here's the data flow:

1. **Pre-routing classifier**: Extract features from the raw request—token count, parse complexity (can you identify specific line ranges?), presence of validation constraints like "must pass type-check."

2. **Route selection**: Choose model tier and context window size. Log the decision with a unique request ID.

3. **Agent execution**: The selected model runs through plan/edit/validate. If validation fails (linter errors, test failures), that's a signal the fast model was insufficient.

4. **Feedback loop**: Failed validations with fast models increment a "complexity score" for similar future requests. This score decays over time as models improve.

Concretely, if a request to "add error handling to this HTTP client" routes to Claude Haiku and produces code that fails your linter, the next similar request automatically routes to GPT-4. The decay means you'll re-test Haiku in a week, capturing model updates without manual intervention.

Embedding routing logic in your application code—rather than relying on a vendor's black-box gateway—gives you control over the criteria that matter for your workload. You decide what "complex" means based on your actual validation outcomes.

Classification without a classification model

Skip the LLM-calls-LLM approach for routing—it doubles your latency and cost. Instead, build a lightweight classifier from structured features you can extract in under 10ms.

**Syntactic complexity**: Run a fast parser (tree-sitter is 1–2ms) and count nesting depth, number of function definitions, and cyclomatic complexity. Requests targeting deeply nested code with high complexity go to the smart model.

**Validation requirements**: If the user includes "must compile" or references CI checks, route to smart. These constraints are explicit in Goatfied's interface, so you're not guessing.

**Historical edit scope**: Track the median diff size for this user's past requests. If they typically make 5-line changes and this request says "restructure the module," route smart. If they're asking for a typo fix, route fast.

**File type heuristics**: Terraform/YAML/JSON changes with no logic modification? Fast model. TypeScript with generic inference or Rust with lifetime annotations? Smart model.

def classify_request(edit_request, repo_context):

    # Fast path: mechanical edits

    if edit_request.type == "lint_fix":

        return Tier.SMALL

    if edit_request.affects_lines < 5 and not repo_context.touches_interfaces:

        return Tier.SMALL

    

    # Escalate if crossing boundaries

    if repo_context.num_files_changed > 3:

        return Tier.FRONTIER

    if edit_request.contains_architecture_keywords():

        return Tier.FRONTIER

    

    # Check retry history

    if edit_request.retry_count > 0:

        return Tier.FRONTIER

    

    # Default to mid-tier

    return Tier.MID

Train a simple decision tree or logistic regression on 500 labeled examples (fast-success, fast-fail-retry, smart-only). You'll hit 85%+ accuracy in a day. The classifier runs in <10ms, adds negligible latency, and you can introspect its decisions in logs.

The fallback cascade that preserves reliability

Hard rule: every fast-model response must flow through the same validation gates you apply to smart-model output. The moment you skip validation "because the cheap model is usually fine," you've opened the door to silent degradation.

Your Tier 2 model will fail. Plan for it. Goatfied's validation gates make this explicit: after every edit, run compile/lint/test. If a Tier 2 attempt fails validation, escalate to Tier 3 automatically. The key is preserving the conversation context—the frontier model needs to see what the mid-tier model tried and where it got stuck.

async function executeWithFallback(request: EditRequest): Promise<Edit> {

    let attempt = await tier2Model.edit(request);

    

    for (let retry = 0; retry < 3; retry++) {

        const validation = await validateEdit(attempt);

        if (validation.passed) return attempt;

        

        attempt = await tier2Model.edit({

            ...request,

            previousAttempt: attempt,

            validationErrors: validation.errors

        });

    }

    

    // Escalate with full context

    return tier3Model.edit({

        ...request,

        tier2Attempts: allAttempts,

        escalationReason: "validation_failure"

    });

}

The failure mode that kills trust: the fast model returns broken code, the user sees it, *then* you re-run with the smart model. Now the user knows your system guessed wrong.

Instead, use **speculative dual execution** for borderline requests. Route to the fast model but *also* queue the smart model in the background. If the fast model's output passes your validation gates (compiles, lints, passes inline tests), return it immediately and cancel the smart-model job. If validation fails, stream the smart model's output without acknowledging the failure.

This adds cost for borderline cases, but typical workloads show 70–80% of requests clearly belong to one tier, leaving only 20–30% as speculative. For those, you're paying 1.5× instead of 2×, and you've eliminated user-facing retries.

Prompt templates and latency budgets per tier

Fast and smart models have different strengths. Don't send identical prompts to both and expect optimal results from either. Fast models work best with tightly scoped instructions: "Fix this linter error on line 42" or "Add type annotations to this function signature." Smart models can handle looser, more exploratory prompts: "Refactor this module to separate IO concerns from business logic" or "Propose three approaches to improve cache hit rates here."

Keep two prompt templates per task type. Route first, then format the prompt for the chosen model.

const fastPrompt = `Fix the TypeScript error on line ${line}: ${error}`;

const smartPrompt = `This module has a type error at line ${line}. 

Context: ${fileContent}

Error: ${error}

Refactor if needed to resolve type safety issues while preserving behavior.`;

Fast models usually respond in 500ms–2s; smart models can take 5s–15s or more, especially with chain-of-thought prompting. Encode latency budgets in your routing logic. For interactive tasks, set a strict timeout on the fast model attempt (say, 3 seconds). If it hasn't responded, either cancel and route to smart, or accept the fast model's partial output if it's streaming. For background tasks, remove the timeout and let the smart model think as long as needed.

Implement tiered timeouts. Fast models should respond in under two seconds; if they don't, cancel and escalate to the smart model immediately. Don't wait for a slow response from a model optimized for speed—it's either stuck or you've hit a pathological input.

Cost and latency trade-offs you can measure

Set budgets at the request level, not the model level. For each incoming request, assign a latency target (p95 < 3s for fast, < 10s for smart) and a cost cap.

Track these metrics per route:

• **Cost per satisfied request**: Total model cost divided by requests that passed validation. A high-cost fast model with 40% retries is worse than a slightly pricier smart model with 95% success.
• **Latency percentiles by intent**: "Explain this code" requests tolerate 10s p95; "fix this lint error" requests don't. Route aggressively for intents with tight latency budgets.
• **Retry rate by classifier confidence**: If your classifier was 90% confident and still routed wrong, your features are missing signal.

For a team running 10,000 requests/week, switching 30% of those from GPT-4 to Haiku (with a 15% retry rate) typically cuts model costs by 50–60% while keeping p95 latency under 4 seconds. The retry overhead is offset by the 3–5× speed and cost difference between tiers.

Cost per task is a vanity metric. What you actually care about: successful edit rate, time to successful edit, and cost per successful edit. Track these by tier and by task type. You'll discover that routing everything through Tier 3 has a 92% success rate and costs $0.40 per edit, while your current strategy has an 89% success rate and costs $0.18 per edit. That 3% difference might represent the edits that ship features versus the ones that waste engineer time on hallucinated code that doesn't compile.

Handling capacity limits and quota exhaustion

Model providers rate-limit aggressively. Your routing logic must account for the scenario where the smart model's quota is exhausted but the fast model still has headroom.

Maintain a circuit breaker per model tier. After N consecutive 429s or 5xx errors from the smart model, automatically downgrade all new requests to the fast model for a cooldown window. Log these downgrades prominently; they're early warning that your traffic scaled beyond your provisioned capacity.

For self-hosted deployments, this looks different. You control the inference hardware, so "quota exhaustion" becomes "GPU memory exhausted" or "request queue saturated." Build admission control: if the smart model's queue depth exceeds a threshold, reject routing decisions that would add to it and send requests to the fast tier instead.

if not smart_model_semaphore.try_acquire(timeout=0.1):

    return route_to_fast_model(request)

Goatfied's managed offering handles this automatically, but self-hosted deployments require explicit queue monitoring and fallback chains.

When to skip routing entirely

Some problem domains don't tolerate probabilistic quality. If you're generating security policies, database migrations, or deployment configs, the cost of a single incorrect output dwarfs the savings from aggressive routing. In these cases, hardcode the smart model and move on.

Similarly, if your validation layer is strong enough to catch 99% of fast-model failures, but retrying those failures with the smart model costs more than just using the smart model initially, the math argues for skipping routing entirely. Calculate your actual retry rate and multiply by the incremental cost of a smart-model call. If that's less than the per-request savings from routing, you're optimizing the wrong thing.

Goatfied defaults to the smart model for edit steps that touch files flagged as high-risk—deployment configs, auth code, database schemas—because the compile-time validation doesn't catch semantic errors in those domains. We route aggressively everywhere else.

Operational hygiene: logging and drift detection

Your routing system will degrade silently as models update, user behavior shifts, and edge cases accumulate. Build observability from day one.

Add structured logging for every model call: which model you chose, why, how long it took, whether validation passed, and the cost.

type RoutingMetric = {

  modelTier: "fast" | "smart";

  routingScore: number;

  validationPassed: boolean;

  retryModel?: "smart";

  retryPassed?: boolean;

  cost: number;

  latencyMs: number;

};

You need dashboards that answer:

• What's the success rate by model tier?
• What's the average cost per successful completion?
• What's the p95 latency by route?
• Which routing rules trigger most often?

**Weekly confusion matrix**: Which classifier predictions led to retries? Retrain your decision tree monthly with fresh data.

**Model drift alerts**: If Haiku's retry rate jumps from 15% to 30% over two weeks, either the model regressed or user requests changed. Investigate before costs spike.

Alert on spikes in cheap-model retry rates or drops in smart-model validation pass rates. Both indicate your routing boundary drifted—either user behavior changed, your prompts changed, or model providers updated their APIs and semantics shifted.

Self-hosted Goatfied deployments make this straightforward: all logs stay in your infrastructure, and you control the feedback loop. Managed deployments expose routing decisions via webhooks, so you can pipe them into your existing observability stack.

Set a weekly review cadence. Look for routing rules that consistently escalate to the smart model, or fast-model attempts that rarely succeed. Adjust thresholds based on measured outcomes, not intuition. If 90% of edits under 50 lines succeed with the fast model, lower the complexity threshold. If retries are frequent for a certain file pattern, route it to the smart model from the start.

Scaling routing complexity only when simple rules plateau

You'll eventually hit diminishing returns on manual thresholds. When that happens—and only then—consider a lightweight ML classifier. Train a small model (logistic regression, gradient-boosted trees, even a fine-tuned embedding similarity check) on your logged routing decisions and their outcomes. Use it to score incoming tasks, and route based on predicted difficulty.

But start simple. Most teams never need ML-based routing. Conditional logic plus validation feedback covers the vast majority of cost savings and quality improvements. Add complexity only when you have data proving the simpler system is leaving money on the table.

The practical implementation: a few conditionals get you 80% of the value. You can measure how often each path is taken, track success rates per route, and adjust thresholds with confidence. Once your routing rules are stable for a month and you have real production data showing where they fail, then consider whether a learned classifier would improve on hand-tuned thresholds.

Related reading

• [Agentic planning for cross-package refactors](/blog/agentic-planning-for-cross-package-refactors)
• [Branch-aware context retrieval pipelines](/blog/branch-aware-context-retrieval-pipelines)
• [Completion ranking with repository signals](/blog/completion-ranking-with-repository-signals)
• [Context window budgeting under token pressure](/blog/context-window-budgeting-under-token-pressure)