Model Routing Between Fast And Smart Engines Production: practical systems guide

Most production AI systems waste money running expensive models on tasks that cheaper ones handle just as well. The difference between a $0.002/1k-token call and a $0.06/1k-token call compounds fast when you're processing hundreds of thousands of requests daily. Smart routing—sending simple queries to fast models and hard problems to capable ones—turns into a cost lever worth engineering properly.

The challenge isn't the concept; it's building routing logic that doesn't add fragility, doesn't require a PhD to tune, and actually improves over time as your workload changes. We'll walk through a practical system that routes between a fast model (GPT-4o-mini, Claude Haiku, or similar) and a smart model (GPT-4, Claude Opus/Sonnet, o1) based on task complexity, maintains acceptable latency, and gives you real data to refine thresholds.

Routing decisions belong in your application layer

Embedding routing logic in your application code—rather than relying on a vendor's black-box gateway—gives you control over the criteria that matter for your workload. You decide what "complex" means: token count, presence of certain keywords, user tier, previous attempt failures, or a lightweight classifier score.

Start with a simple decision tree. For a coding assistant like Goatfied, you might route based on:

• **Edit scope**: single-line fixes or doc updates → fast model; multi-file refactors or architectural changes → smart model
• **Compilation state**: if the plan already compiled and passed lint → fast model for minor tweaks; if validation failed → smart model to rethink approach
• **Retry count**: first attempt → fast model; second attempt after a failure → smart model

This doesn't require ML. A few conditionals get you 80% of the value. You can measure how often each path is taken, track success rates per route, and adjust thresholds with confidence.

function selectModel(task: Task): ModelTier {

  if (task.retryCount > 0) return 'smart';

  if (task.editScope === 'multi-file') return 'smart';

  if (task.estimatedComplexity > 7) return 'smart';

  return 'fast';

}

Use validation gates as routing feedback

In Goatfied's agent loop (plan → constrain → edit → validate → retry), every edit hits compile, lint, and test gates before shipping. These gates give you ground truth: did the model's output actually work? If the fast model's edit fails validation, retry with the smart model. If it passes, you saved money without sacrificing quality.

This creates a natural feedback loop. Track which tasks the fast model handles successfully on first attempt versus which need escalation. Over time, patterns emerge—maybe certain file types, certain error messages, or certain prompt structures correlate with fast-model failure. Use that data to refine your routing logic.

The validation step also bounds your risk. A bad fast-model output costs you one retry, not a broken deployment. The smart model acts as a safety net, not a fallback you hope never triggers.

Maintain separate prompt templates per tier

Fast and smart models have different strengths. Don't send identical prompts to both and expect optimal results from either. Fast models work best with tightly scoped instructions: "Fix this linter error on line 42" or "Add type annotations to this function signature." Smart models can handle looser, more exploratory prompts: "Refactor this module to separate IO concerns from business logic" or "Propose three approaches to improve cache hit rates here."

Keep two prompt templates per task type. Route first, then format the prompt for the chosen model. This also makes A/B testing cleaner—you can compare smart-model performance on terse prompts versus verbose ones without conflating model choice and prompt style.

const fastPrompt = `Fix the TypeScript error on line ${line}: ${error}`;

const smartPrompt = `This module has a type error at line ${line}. 

Context: ${fileContent}

Error: ${error}

Refactor if needed to resolve type safety issues while preserving behavior.`;

Instrument everything, optimize thresholds from data

Add structured logging for every model call: which model you chose, why, how long it took, whether validation passed, and the cost. Export this to your observability stack (Datadog, Grafana, CloudWatch, wherever you already look). You want dashboards that answer:

• What's the success rate by model tier?
• What's the average cost per successful completion?
• What's the p95 latency by route?
• Which routing rules trigger most often?

Set a weekly review cadence. Look for routing rules that consistently escalate to the smart model, or fast-model attempts that rarely succeed. Adjust thresholds based on measured outcomes, not intuition. If 90% of edits under 50 lines succeed with the fast model, lower the complexity threshold. If retries are frequent for a certain file pattern, route it to the smart model from the start.

Cost per successful task matters more than cost per call. A fast model that fails 40% of the time and requires smart-model retries is more expensive than routing to the smart model initially.

Handle latency budgets explicitly

Fast models usually respond in 500ms–2s; smart models can take 5s–15s or more, especially with chain-of-thought prompting. If your users are waiting in an IDE for an edit, 15 seconds feels like forever. If you're batch-processing pull requests overnight, it's irrelevant.

Encode latency budgets in your routing logic. For interactive tasks, set a strict timeout on the fast model attempt (say, 3 seconds). If it hasn't responded, either cancel and route to smart, or accept the fast model's partial output if it's streaming. For background tasks, remove the timeout and let the smart model think as long as needed.

Goatfied's compile-first approach helps here: if you're running edits through validation gates anyway, you can afford to optimize for correctness over speed in non-interactive flows. Users expect the CI loop to take minutes; they don't expect their inline autocomplete to lag.

Scale routing complexity only when simple rules plateau

You'll eventually hit diminishing returns on manual thresholds. When that happens—and only then—consider a lightweight ML classifier. Train a small model (logistic regression, gradient-boosted trees, even a fine-tuned embedding similarity check) on your logged routing decisions and their outcomes. Use it to score incoming tasks, and route based on predicted difficulty.

But start simple. Most teams never need ML-based routing. Conditional logic plus validation feedback covers the vast majority of cost savings and quality improvements. Add complexity only when you have data proving the simpler system is leaving money on the table.

Related reading

• [Self-hosted AI coding assistants: a complete Goatfied deployment guide](/blog/self-hosted-ai-coding-assistants-goatfied-deployment-guide)
• [Deployment playbook #2: practical Goatfied tactics for shipping PRs](/blog/goatfied-deployment-playbook-2)