Why Smaller Models Win

The Default Assumption

When a developer faces a hard problem, the instinct is to reach for the biggest model available. Claude Opus. GPT-5.5. The frontier. If the task is important, you want the best tool.

This instinct is reasonable for a single, well-scoped question. It breaks down badly for complex, multi-step missions — the kind of work that defines real agent workflows.

The assumption that "bigger model = better results" is one of the most expensive mistakes in AI agent development today. Not because frontier models are bad. Because complex missions are not single tasks. They are collections of sub-tasks, most of which do not require frontier-level capability — and routing all of them through a $15-per-million-token model when a $0.15-per-million-token model would do the job is not just wasteful. It actively produces worse outcomes.

This article explains why, and what to do instead.

Why Monolithic Frontier Execution Fails Complex Missions

Context Window Saturation

A complex mission — build a feature, run a GTM campaign, refactor a codebase — generates a lot of context. Requirements, prior decisions, code state, research, intermediate outputs. When you route the entire mission through a single model session, all of that context competes for attention in a single context window.

Frontier models have large context windows, but attention is not uniform across them. Research consistently shows that models perform worse on information buried in the middle of long contexts — the "lost in the middle" problem. A 200,000-token context window does not mean 200,000 tokens of equally reliable reasoning. It means the model is doing its best with a lot of noise.

Decomposing the mission into sub-tasks and routing each one with only the context it needs eliminates this problem. Each sub-task gets a clean, focused prompt. The model is not fighting through irrelevant context to find the signal.

Cost Compounding

GPT-5.5 is priced at $5 per million input tokens and $30 per million output tokens for standard usage, rising to $10/$45 for long-context requests. Claude Opus is in a similar range. These are the right prices for the hardest reasoning tasks. They are the wrong prices for boilerplate generation, formatting, summarization, classification, and the dozens of other sub-tasks that make up a real mission.

A typical complex mission might involve: writing a technical spec (hard reasoning, frontier-appropriate), generating unit tests for that spec (mechanical, small-model-appropriate), summarizing research (moderate, mid-tier-appropriate), writing a changelog entry (trivial, smallest-model-appropriate), and reviewing a diff for obvious errors (moderate, mid-tier-appropriate).

Routing all five through a frontier model at $30/M output tokens is not a quality decision. It is a habit. And it compounds across every mission you run.

Single Point of Failure

When a monolithic frontier session fails — hits a rate limit, produces a hallucination, loses track of a constraint mid-context — the entire mission fails. There is no isolation. You cannot tell which sub-task went wrong. You restart from scratch or spend time debugging a 50,000-token context to find the error.

Decomposed, routed workflows fail at the sub-task level. The failure is isolated, visible, and recoverable. You retry the specific sub-task that failed, with the specific context it needs, without touching the rest of the mission.

No Specialization

Frontier models are generalists. They are trained to be good at everything, which means they are not optimized for anything specific. A model fine-tuned for code generation — Qwen 2.5 Coder 32B scores 80.1% on HumanEval, DeepSeek Coder V2 33B scores 83.5% — can outperform a larger generalist model on coding sub-tasks while running locally at zero marginal cost. Mistral Small 3.2 achieves 92.9% Pass@5 on HumanEval Plus. These are not consolation prizes. They are purpose-built tools that beat the generalist on their home turf.

The Ensembling Insight

The core insight is simple: decompose → route → synthesize produces better results than monolithic execution.

Break the mission into sub-tasks. Route each sub-task to the model best suited for it — by capability, cost, latency, or trust profile. Synthesize the outputs into a coherent result.

This is not a new idea in software engineering. Microservices beat monoliths for complex systems because specialization, isolation, and independent scaling produce better outcomes than trying to make one thing do everything. The same logic applies to AI agent workflows.

The challenge has always been the coordination overhead. Decomposing a mission manually, deciding which model handles which sub-task, managing the routing logic, and synthesizing outputs is more work than just sending a prompt to Claude. That overhead is why developers default to the monolith — not because it produces better results, but because it is easier.

The right answer is to make the coordination automatic.

Four Practical Ensembling Patterns

1. Tiered Routing by Task Complexity

The simplest pattern: classify each sub-task by complexity and route to the appropriate model tier.

Hard reasoning, novel problem-solving, and high-stakes decisions go to frontier models. Standard coding tasks, structured generation, and moderate reasoning go to mid-tier models (Qwen 3.5 8B, Mistral Small 3.2). Boilerplate, formatting, classification, and summarization go to the fastest, cheapest model available — often a local model with zero marginal cost.

The classification itself can be automated. A lightweight model can assess sub-task complexity and assign a tier before routing. The overhead is minimal; the cost savings are substantial.

2. Parallel Perspectives

For high-stakes sub-tasks where you want confidence in the output, route the same sub-task to two different models simultaneously and synthesize their responses.

This is particularly valuable for code review, security analysis, and architectural decisions — cases where a single model's blind spots could be costly. Two models with different training distributions will surface different issues. A synthesis step that identifies agreement and flags divergence gives you a more reliable output than either model alone.

The cost is roughly double for that sub-task. For genuinely high-stakes decisions, it is usually worth it.

3. Fan-Out Decomposition

For missions with parallel workstreams — a product launch that requires simultaneous work on copy, code, and research — fan out the sub-tasks across multiple model instances running in parallel.

Instead of a single model working through the mission sequentially, multiple models work on independent sub-tasks simultaneously. Total wall-clock time drops dramatically. Each model works with only the context relevant to its sub-task. The outputs are synthesized at the end.

This pattern is particularly effective for missions where the sub-tasks are genuinely independent — where the output of one does not gate the start of another.

4. Ensemble Voting for High-Stakes Decisions

For binary decisions with significant consequences — "should we merge this PR," "does this code have a security vulnerability," "is this copy compliant with our brand guidelines" — route the decision to three models and take the majority vote.

Ensemble voting reduces the impact of any single model's errors or biases. It is more expensive than a single model call, but for decisions where a wrong answer has real downstream cost, the reliability improvement justifies it.

The Cost Math

Here is a concrete example. A mission: "Implement a new authentication flow, write tests, update the docs, and draft the changelog."

Monolithic frontier execution:

• All sub-tasks routed to GPT-5.5 at $5/$30 per million tokens
• Estimated total tokens: 180,000 input, 40,000 output
• Cost: $0.90 input + $1.20 output = $2.10

Routed multi-model execution:

• Architecture design (frontier, Claude Opus): 20,000 input / 8,000 output → $0.24
• Code implementation (Qwen 3.5 8B, local): $0.00
• Unit test generation (Mistral Small 3.2 via API at $0.10/$0.30 per million): 30,000 input / 12,000 output → $0.007
• Documentation update (mid-tier model at $0.15/$0.60 per million): 15,000 input / 5,000 output → $0.005
• Changelog draft (smallest tier at $0.05/$0.15 per million): 5,000 input / 1,000 output → $0.0004
• Total: ~$0.25

The routed run costs roughly 88% less. More importantly, the architecture decision — the sub-task that actually benefits from frontier reasoning — still gets frontier reasoning. The mechanical sub-tasks get fast, cheap, purpose-built models. The quality ceiling is the same or higher. The cost floor is dramatically lower.

These numbers will vary by mission. The ratio holds.

The Honest Caveat

Ensembling adds coordination overhead. For simple, single-step tasks — "summarize this document," "write a regex for this pattern," "explain this error message" — the decomposition and routing logic costs more than it saves. The monolith wins for simple tasks.

The crossover point is roughly where a task requires more than two or three sequential steps, involves parallel workstreams, or has sub-tasks with meaningfully different complexity profiles. Below that threshold, just use the best model for the job and move on.

The goal is not to ensemble everything. It is to stop defaulting to monolithic execution for complex missions where ensembling consistently produces better outcomes.

How Medley Implements This Automatically

The reason most developers do not ensemble is not that they disagree with the logic. It is that the coordination overhead is real and manual decomposition is tedious.

Medley's routing layer makes this automatic. You describe the mission. Medley decomposes it into a DAG of sub-tasks, assigns each sub-task to the appropriate model tier based on complexity and cost optimization, and runs the workflow. The decomposition is visible and editable — you can override any routing decision before execution.

After the mission completes, Medley shows you a cost receipt: each sub-task, the model that handled it, the token count, and the cost. Not an aggregate bill — a per-task breakdown that tells you exactly where your spend went and why each model was chosen.

This is what "mission control for your AI agents" means in practice. Not a chat interface. Not a session manager. A system that takes the mission, produces the plan, routes the work, and shows you the result — with full visibility into every decision along the way.

If you are running complex agent workflows today and want to stop paying frontier prices for boilerplate work, Medley is worth a look: medley.sh