Routing between fast and smart models without users noticing

Every request to an LLM-powered feature hits the same fork in the road: route it to a fast, cheap model and risk a mediocre answer, or send it to a slow, expensive model and make the user wait. Most products pick one model per feature and call it a day. The sharper approach is LLM model routing—dynamically choosing which model handles each request based on complexity, context, and cost constraints—so users get answers that feel instant when possible and thorough when necessary, without ever seeing the seams.

Routing works because not every prompt needs GPT-4 or Claude Opus. A user asking "What does this error mean?" in a tight loop during a debugging session can usually be satisfied by a 7B parameter model running locally or a distilled variant in the cloud. A user crafting a complex SQL migration from a vague English description needs the reasoning depth of a frontier model. The trick is deciding which is which before the model runs, then falling back gracefully when the cheaper choice fails.

Signals that predict when to route up

The most reliable routing signal is syntactic complexity of the prompt itself. Count the number of context items (files, stack frames, docs pages), the length of the user's natural language instruction, and whether the request includes negations, conditionals, or multi-step logic. A prompt like "add logging" with one file in context can go to a small model. A prompt like "refactor this to use dependency injection but preserve the existing API surface and don't break the tests" with twelve files is a strong candidate for a large model.

Stated user intent also matters when you can infer it. If a user is in a tight edit-compile-test cycle—making small changes, running tests, iterating—they value speed over perfection. If they explicitly open a "deep refactor" or "design review" mode, they're signaling they'll tolerate latency for quality. Goatfied's agent loop tracks how many plan–edit–validate cycles have run in the current session; after three failed attempts on a small model, the router escalates to a larger one before the user asks.

Historical success rates per task type give you a third axis. Log every routing decision, the chosen model, and whether the output passed your validation gates (compilation, linting, tests). If "generate a new React component from scratch" succeeds 40% of the time on your 7B model but 92% on your 70B model, bias future routing toward the larger model for that pattern. If "fix this typo" succeeds 98% on the small model, keep it there.

Hybrid routing: classifier models and heuristics

You can build a dedicated classifier model that predicts task difficulty. Train a lightweight model (even logistic regression works) on features like prompt length, number of code tokens in context, presence of keywords ("refactor," "optimize," "bug"), and user history. The classifier outputs a complexity score from 0 to 1; you set thresholds—say, below 0.3 goes to the small model, above 0.7 to the large, and the middle band to a medium-tier option.

In practice, a hybrid of heuristics and ML often beats pure ML alone. Start with rules:

def route_request(prompt, context_files, user_session):

    if len(context_files) > 8:

        return "large"

    if "refactor" in prompt.lower() or "redesign" in prompt.lower():

        return "large"

    if user_session.recent_failures > 2:

        return "large"

    if len(prompt.split()) < 15 and len(context_files) <= 2:

        return "small"

    return "medium"

Then layer on a classifier for the ambiguous middle cases. This keeps routing explainable and debuggable while still learning from data.

Cascading requests: try small, retry large

Instead of picking one model upfront, cascade: send the request to the small model, run your validation gates (compile, lint, test), and if it fails, immediately retry with the large model. The user sees the latency of the large model only when the small model wasn't good enough, and you save cost on every successful small-model run.

Cascading works well when your validation is fast. Goatfied runs TypeScript compilation and ESLint in under two seconds for most codebases. If the small model generates code that doesn't compile, the validation failure triggers a retry with a larger model, and the total round-trip is still faster than many products' single large-model pass because the large model only runs when needed.

The downside is doubled latency on failures. If your small model succeeds 50% of the time, half your users wait for two model calls instead of one. Mitigate this by tuning your routing thresholds so the small model only handles requests where it succeeds >80%, or by running the large model speculatively in parallel and canceling it if the small model succeeds first (burns cost but saves user time).

Parallel voting for high-stakes requests

For requests where correctness matters more than latency—production deploys, schema migrations, security-sensitive changes—send the same prompt to two or three models in parallel and compare outputs. If all models agree, ship the result. If they diverge, either surface both options to the user or route to an even larger model as a tiebreaker.

Parallel voting catches hallucinations and subtle logic errors that a single model might miss. It roughly triples your inference cost for those requests, so reserve it for high-value or high-risk operations. Goatfied uses parallel voting when a user explicitly requests a "confident" generation or when the diff touches files flagged as critical in the repo config.

Prompt rewriting to lift small models

Sometimes a small model fails not because the task is hard, but because the prompt is ambiguous or lacks structure. Before routing to a large model, try rewriting the prompt with more specificity or breaking it into sub-prompts.

For example, if a user asks "improve this function," a small model might flail. Rewrite it to "Add input validation, extract the database query into a helper, and add JSDoc comments" and the same small model often succeeds. You can use a cheap large model to do the rewriting (meta-prompting) or a heuristic library of common rewrites per task type.

This adds a round-trip but keeps you on the small model, which is usually a net win on latency and cost if the rewrite is cached or batched.

Handling model version drift

Models get updated, deprecated, and replaced. Your routing logic needs to survive these changes without manual rewrites. Store model identifiers as aliases—"small", "medium", "large"—and map them to actual model names (gpt-4o-mini, claude-3.5-sonnet, etc.) in a config file. When a provider deprecates a model, update the alias mapping once instead of touching every routing callsite.

Track success rates per model version, not just per model family. If gpt-4o-2024-11-20 performs worse on TypeScript generation than gpt-4o-2024-08-06, your historical routing data will show it and you can roll back or adjust thresholds.

User-facing latency vs. backend cost

Routing optimizes for two conflicting goals: minimize cost and minimize user-perceived latency. The equilibrium depends on your business model. If you charge per request, bias toward small models and accept occasional quality degradation. If you charge a flat monthly fee, bias toward large models to maximize user satisfaction. If you're self-hosted, the user controls the tradeoff directly—let them configure routing thresholds in their instance settings.

Goatfied's managed offering defaults to aggressive small-model routing with cascading retries because most tasks (fixing typos, adding log statements, renaming variables) genuinely don't need frontier models. Self-hosted customers often flip the defaults to prefer large local models because they've already paid the GPU cost and want maximum quality.

Measuring routing effectiveness

You can't improve what you don't measure. Log every routing decision with the prompt hash, chosen model, latency, cost, and validation result. Build dashboards that show:

• Success rate by model and task type
• P50/P95 latency by routing path (small-only, small-then-large cascade, large-only)
• Cost per successful request by routing strategy
• User retry rate (if users regenerate after a small-model answer, your routing is too aggressive)

Run A/B tests on routing thresholds. If bumping the small-model complexity threshold from 0.3 to 0.4 increases failures by 5% but cuts cost by 18%, that might be a good trade. If it doubles user retries, it's not.

Related reading

• When fine-tuning beats RAG for code, and when it doesn't
• Quantization tradeoffs for code models: what you actually lose