Syntic
All posts

Published October 4, 2026 in Models

Why One AI Model Is Not Enough

By Syntic AI

On 7 August 2025, OpenAI launched GPT-5, and the flagship was not a single model. Its system card describes "a unified system" with a fast model that answers most questions, a deeper reasoning model for harder ones, and "a real-time router" that decides which to use based on conversation type, complexity, tool needs and explicit intent.

The next day, Sam Altman explained why many users had found the launch disappointing: "Yesterday, we had a sev and the autoswitcher was out of commission for a chunk of the day, and the result was GPT-5 seemed way dumber." The models were fine. The part that chose between them had failed.

That episode is a compact version of where AI systems are heading. The question used to be which model to buy. For serious deployments it is becoming a different question: what policy decides which model handles each piece of work, who checks the result, and what happens when a model is slow, wrong, retired or unavailable. Quality, cost and reliability are increasingly set by that policy, not by any single model in it.

The price spread is the starting point

The economic case starts with a simple observation. When Stanford researchers Lingjiao Chen, Matei Zaharia and James Zou surveyed commercial model APIs in 2023 for the FrugalGPT paper, they found fees that differed "by two orders of magnitude." A password reset or a field extraction does not call for the same compute as a contract analysis.

FrugalGPT tested a cascade: send a query to a cheap model first, score the answer, and escalate to a more expensive model only when the score falls short. On the authors' benchmarks, the cascade matched the best individual model with up to 98% lower cost, or improved accuracy by 4% at the same cost. The numbers reflect 2023 prices and datasets; the structure has held up.

Routing and cascading are different bets

A cascade tries the small model and escalates on doubt. It pays for both calls on hard queries, and it needs a reliable way to judge whether the first answer is good enough.

A router looks at the request before any model runs and sends it to one place. It pays once, but it must predict difficulty from the input alone.

Microsoft and University of British Columbia researchers took the router approach in "Hybrid LLM" (ICLR 2024), predicting query difficulty and sending easy queries to a small model. They reported up to 40% fewer calls to the large model with no drop in response quality. RouteLLM, from the LMSYS and Berkeley group, trained routers on human preference data to choose between GPT-4 Turbo and Mixtral 8x7B. While keeping 95% of GPT-4's performance, it cut costs by more than 85% on MT-Bench, 45% on MMLU and 35% on GSM8K.

The spread between those three numbers is the lesson: savings depend on the traffic. Nobody can quote a routing saving without knowing the request mix.

Routers are models, and they fail like models

The RouteLLM team reported a detail that deserves more attention than the headline savings. Routers trained only on their Chatbot Arena preference data performed "at a near-random level" on MMLU, because most of those questions were out of distribution. Performance improved sharply once they added about 1,500 labelled examples from the target domain, less than 2% of the training data.

A router is a classifier, and it inherits every weakness of classifiers: it is accurate on traffic that resembles its training data and fragile on traffic that does not. When it errs toward the small model, quality drops silently, which is exactly what GPT-5 users noticed on launch day. When it errs toward the large model, the bill grows and nobody notices until month end.

OpenAI's description of the GPT-5 router acknowledges the need for feedback: the system card says it is "continuously trained on real signals, including when users switch models, preference rates for responses, and measured correctness." Any router needs the same: labelled outcomes from its own traffic, retraining as the mix changes, and a record of which path each request took. Public benchmarks such as RouterBench, with more than 405,000 inference outcomes, compare routing methods; they cannot say how a team's own customers phrase things.

Specialists do work a generalist should not

Most production systems already include models that are not chat models at all.

Speech recognition and speech synthesis turn calls into text and back. Embedding models turn documents into vectors for retrieval. Safety classifiers screen inputs and outputs: Meta's Llama Guard, for example, is a 7-billion-parameter model built to classify both user prompts and model responses against a safety taxonomy. Each is cheaper and faster than a frontier model for its narrow task, and often more predictable.

Some of these boundaries are dissolving. As multimodal models learn to listen and look directly, a separate transcription step can disappear from some pipelines. But voice systems still face hard latency budgets, as described in how the call center is becoming software, and a small, specialised model is often the only thing fast enough at a given point in the conversation.

Even a single model's serving stack can involve more than one model. Speculative decoding, introduced by Google researchers at ICML 2023, uses a small draft model to propose several tokens that the large model then checks in parallel. On T5-XXL it delivered a 2 to 3 times speed-up with outputs identical to the large model's. The user sees one model; underneath, two are working together.

A second model as the checker

The most powerful reason to use more than one model may be verification rather than cost.

In 2021, OpenAI researchers trained a "verifier" to score candidate solutions to grade-school maths problems, generating many answers and keeping the one the verifier ranked highest. A 6-billion-parameter verifier slightly outperformed a fine-tuned 175-billion-parameter model, a gain the authors equated to roughly a 30-fold increase in model size. Checking an answer can be easier than producing one, and that asymmetry can be bought with a separate model.

Language models used as judges show the same promise with caveats attached. The MT-Bench study (NeurIPS 2023) found that GPT-4 as a judge agreed with human preferences more than 80% of the time, about the rate at which humans agree with each other. The same paper catalogued position bias, verbosity bias and self-enhancement bias, where a judge favours answers that resemble its own.

That last bias points to a deeper problem. Ensembles and judges help only if their errors are independent. A 2025 ICML study by Elliot Kim and colleagues evaluated more than 350 models and found that on one leaderboard dataset, two models agreed on the same wrong answer 60% of the time when both were wrong. Larger and more accurate models had more correlated errors, even across different architectures and providers. Asking a second frontier model to check the first gives less independent assurance than it appears to. Verification works best when the checker has access to something the generator lacked: a test suite, a database lookup, a calculation, a rule, or a person.

A model is a dependency

Teams that build on a single model also inherit its operational risks.

APIs refuse requests. Anthropic's documentation, for example, lists a 429 error when an organisation hits a rate limit and a 529 error when "the API is temporarily overloaded", which can happen during high traffic across all users. Retrying the same endpoint helps with a blip, not with a sustained outage.

Models are also retired. The same provider commits to at least 60 days' notice before retiring a publicly released model, after which "requests to retired models will fail." Any team with a single hard-coded model name is running a migration project on someone else's schedule.

The subtler risk is degradation without downtime. In September 2025, Anthropic published a postmortem describing three infrastructure bugs that degraded some Claude responses from August into September. One was itself a routing error: some requests were sent to servers configured for a different context window and at the worst point, on 31 August, about 16% of Sonnet 4 requests were affected. The company serves its models across three hardware platforms and noted that it "relied too heavily on noisy evaluations" to catch the problem. The model was available the whole time. It was simply, sometimes, worse.

A fallback design therefore needs more than a second endpoint. It needs a pre-tested alternative for each critical task, health checks that measure output quality rather than only uptime, and a decision about when degraded service is better than no service.

Routing for reasons that are not quality

Some routing decisions have nothing to do with accuracy or cost. They follow the data.

Cloud platforms now make this explicit. Amazon Bedrock offers "geographic" inference profiles that keep processing within a region such as the US, EU or Asia-Pacific, and "global" profiles that can use any supported commercial region for roughly 10% lower cost. AWS's own guidance is to choose geographic profiles for compliance requirements. The same trade-off appears elsewhere: a request containing health records or customer financial data may need to go to a model running in a particular jurisdiction, or to a self-hosted model that never leaves the company's network, even if a better model is available elsewhere.

This makes the router a policy enforcement point. It sees every request before any model does, which makes it the natural place to classify sensitivity, strip identifiers, apply residency rules and log the decision. It also makes it a single point of failure for compliance: a routing bug can become a data-transfer incident.

What the system costs to run

None of this is free. Multi-model systems trade per-request savings for engineering overhead.

Evaluation multiplies. A single-model application needs one evaluation suite. A routed system needs evaluations per path, for the router itself, and for the fallback combinations that only appear during incidents. Each prompt may need tuning per model, because models respond differently to the same instructions.

Behaviour becomes harder to explain. When a customer reports a bad answer, the first question is which model produced it. Without a trace that records the routing decision, the model and version, the prompt and the verifier's score, the question cannot be answered. Savings that cannot be measured per path are guesses.

Consistency suffers. Two models can give the same user different tones, formats or refusals within one conversation. For a workflow that writes to business systems, as discussed in AI workflows as the glue between business tools, output schemas need to be validated after every model step, whichever model ran it.

The economics move. As the cost of intelligence falls, the savings from routing shrink in absolute terms on some workloads while the engineering cost stays. A team routing to save money on a cheap workload may find the router costs more to maintain than it saves. Routing for latency, verification, resilience or data governance holds up better than routing for price alone.

For many small deployments, one capable model with a tested fallback and a clear evaluation set is the right design. The multi-model architecture earns its complexity at volume, under strict latency budgets, where errors are expensive, or where data rules require it.

The policy is the asset

The models in any stack will change: prices fall, versions retire, leaderboards reshuffle. What persists is the knowledge of how a particular workload behaves: which requests are easy, which are dangerous, which checks catch real errors, and which fallback produces acceptable results when the primary path fails.

That knowledge lives in the routing policy, the evaluation sets and the traces, not in the model weights. Treat the router as a product with its own tests, owners and release process, and swapping a model becomes routine. Treat it as configuration, and the next launch-day "sev" will look like the model got dumber, when the real problem is that nobody was watching the switch. That is part of why intelligence is becoming infrastructure: infrastructure is judged by how it behaves when components fail.

Sources