Model Routing Between Fast And Smart Engines Operational: practical systems guide

# Model Routing Between Fast And Smart Engines Operational: practical systems guide

Most production AI coding systems waste 70% of their inference budget on tasks that don't need frontier models. A junior developer doesn't call a staff engineer to fix a typo—your routing layer shouldn't either. The difference between a $40/day LLM bill and a $400/day one often comes down to whether you can reliably send syntax fixes to GPT-4o-mini and architecture decisions to Claude Opus.

Building that routing layer is straightforward in principle: check the request, pick a model, send it through. In practice, you'll hit three hard problems immediately: how to classify requests accurately without adding 200ms of latency, how to handle the cases where your fast model fails, and how to measure whether your router is actually saving money or just degrading quality.

The three-tier model strategy

Start with three tiers, not two. Most teams try fast/smart and discover they need a middle ground within a week.

**Tier 1 (sub-200ms)**: Small, locally-hostable models for mechanical edits. Renaming variables, fixing linting errors, applying formatting rules, generating boilerplate from templates. These are the requests where any frontier model is overkill. If you're using Goatfied's agent loop, these are the tasks that hit the constraint phase and fail immediately—perfect candidates for a model that costs 100x less.

**Tier 2 (2-5 second)**: Mid-size models like GPT-4o-mini or Claude Haiku for standard feature work. Implementing a well-specified function, writing tests for existing code, refactoring a class that doesn't touch architectural boundaries. These succeed 85%+ of the time if your prompt includes sufficient context.

**Tier 3 (10-30 second)**: Frontier models for ambiguous or cross-cutting work. Resolving architectural tradeoffs, debugging race conditions, designing new abstractions. Save these for requests where the cost of getting it wrong exceeds the inference cost by orders of magnitude.

The mistake is trying to be clever about which Tier 2 model to use. Pick one and stick with it for a month. Your routing logic needs production data to improve, and A/B testing models simultaneously destroys your ability to learn.

Classification without a classification model

The naive approach is to use an LLM to classify the request itself. This adds latency, cost, and a new failure mode. Instead, build a rule-based classifier from features you can extract in under 10ms:

def classify_request(edit_request, repo_context):

    # Fast path: mechanical edits

    if edit_request.type == "lint_fix":

        return Tier.SMALL

    if edit_request.affects_lines < 5 and not repo_context.touches_interfaces:

        return Tier.SMALL

    

    # Escalate if crossing boundaries

    if repo_context.num_files_changed > 3:

        return Tier.FRONTIER

    if edit_request.contains_architecture_keywords():

        return Tier.FRONTIER

    

    # Default to mid-tier

    return Tier.MID

You need five signals: edit scope (lines/files), whether the change touches public interfaces, whether the request contains architectural language ("design", "should we", "tradeoff"), how many compilation/test failures occurred in the last attempt, and the user's explicit tier preference if they set one.

That last one matters more than you'd expect. Engineers learn which requests need big models. Let them override your router and log those overrides—they're your training data for next month's classifier improvements.

The fallback cascade

Your Tier 2 model will fail. Plan for it.

Goatfied's validation gates make this explicit: after every edit, run compile/lint/test. If a Tier 2 attempt fails validation three times, escalate to Tier 3 automatically. The key is preserving the conversation context—the frontier model needs to see what the mid-tier model tried and where it got stuck.

async function executeWithFallback(request: EditRequest): Promise<Edit> {

    let attempt = await tier2Model.edit(request);

    

    for (let retry = 0; retry < 3; retry++) {

        const validation = await validateEdit(attempt);

        if (validation.passed) return attempt;

        

        attempt = await tier2Model.edit({

            ...request,

            previousAttempt: attempt,

            validationErrors: validation.errors

        });

    }

    

    // Escalate with full context

    return tier3Model.edit({

        ...request,

        tier2Attempts: allAttempts,

        escalationReason: "validation_failure"

    });

}

Log every escalation with its reason. After two weeks you'll see patterns: "Tier 2 fails on async/await refactoring 90% of the time" or "database schema changes always escalate." Update your classifier to route those directly to Tier 3.

Measuring what matters

Cost per task is a vanity metric. What you actually care about: successful edit rate, time to successful edit, and cost per successful edit.

Track these by tier and by task type. You'll discover that routing everything through Tier 3 has a 92% success rate and costs $0.40 per edit, while your current strategy has an 89% success rate and costs $0.18 per edit. That 3% difference might represent the edits that ship features versus the ones that waste engineer time on hallucinated code that doesn't compile.

The Goatfied loop makes this measurement cleaner because validation is mandatory. You know exactly when an edit succeeded (passed compile/lint/test) versus when it burned tokens on invalid output. If you're self-hosting, you have the inference logs to calculate exact cost per token by model.

Set a weekly review: export your router decisions, success rates by tier, and escalation reasons. Adjust the classifier thresholds based on whether you're optimizing for cost or quality. During a fundraising sprint, bias toward Tier 3. During a cost-cutting quarter, tighten the escalation criteria.

The temperature question

Should you run different temperatures by tier? No—at least not initially.

Keep temperature at 0.3 across all tiers. The reliability you gain from deterministic outputs outweighs the creativity you might get from higher temperatures, especially when you're measuring routing effectiveness. Changing temperature and routing strategy simultaneously makes it impossible to know what caused a quality change.

Once your router is stable for a month, then experiment with temperature on Tier 3 only. Even then, you're probably better off improving your prompts.

Related reading

• [Self-hosted AI coding assistants: a complete Goatfied deployment guide](/blog/self-hosted-ai-coding-assistants-goatfied-deployment-guide)
• [Deployment playbook #2: practical Goatfied tactics for shipping PRs](/blog/goatfied-deployment-playbook-2)