Small Models Are Underrated

The reflex to go big

There is a gravitational pull in AI toward the biggest, smartest, most expensive model available. It feels safe. Nobody gets criticized for choosing the frontier flagship, the same way nobody used to get fired for buying from the dominant vendor. So teams route every request — the trivial and the profound alike — to the most powerful model they can find, and pat themselves on the back for not cutting corners.

It is, quite often, a waste. Small models — the lighter, cheaper, faster tier that every major lab and the open-weight community now ship — are dramatically more capable than their reputation suggests, and for a huge fraction of real tasks they are not just "good enough" but genuinely the better engineering choice. Overlooking them costs money, speed, and sometimes privacy, for no gain in quality.

What we mean by "small"

"Small" is relative and keeps shifting, but the idea is stable: models with far fewer parameters than the frontier flagships, designed to run cheaper and faster — sometimes small enough to run on a single machine, a laptop, even a phone. They come from two directions: the efficient tiers offered by the big API providers, and the open-weight models you can download and run yourself with tools like Ollama or LM Studio.

The reason they have become so interesting is that the whole capability curve has risen. A small model today often matches what a flagship could do eighteen months ago. The question is no longer "is the small model smart enough to be useful?" — it usually is — but "is this specific task hard enough to actually need the big one?"

Where small models win

For a surprising range of work, a small model is not a compromise. It is the right tool.

• Classification and routing. Deciding which category a message belongs to, or which path a request should take, is well within a small model's reach — and you may run it on every request, so cost and speed matter enormously.
• Extraction and structuring. Pulling fields out of text, turning messy input into clean structured data, tagging and labeling — bread-and-butter tasks that small models handle crisply.
• Simple rewrites and formatting. Fixing tone, reformatting, summarizing short passages, basic translation — high volume, modest difficulty, perfect for a small model.
• Well-scoped, repetitive jobs. Anything narrow and clearly defined, especially at scale, is a strong fit.

The unifying trait: the task is bounded and the difficulty is moderate. Reserve the frontier model for genuine reasoning, complex multi-step problems, and the long-tail hard cases — the work that actually stretches a model.

The advantages that compound

Choosing small is not just about a lower bill, though the savings are real and often an order of magnitude. Three benefits stack up.

• Cost. Small models can be tens of times cheaper per request. At scale, that is the difference between a feature that is economically viable and one that is not.
• Speed. Fewer parameters means faster responses. For anything interactive, the snappier feel of a small model is a genuine product improvement, not just an internal metric.
• Privacy and control. A small open model can run on your own hardware — even offline — so sensitive data never leaves your environment. For regulated or privacy-critical work, that alone can be decisive.

Bigger is not better; appropriate is better. The skill is matching the model to the difficulty of the job, not defaulting to maximum power for everything.

A practical pattern: prototype big, then downgrade

You do not have to choose perfectly up front. The most effective teams use a simple, deliberate workflow.

1. Prototype on a strong model. Get the feature working first, without worrying about cost. Prove the value.
2. Then try to downgrade. Once it works, test progressively smaller models on the same task, using your eval set to watch quality. You are looking for the smallest model that still clears your bar.
3. Route by difficulty. For the best of both worlds, send the easy majority of requests to a small model and escalate only the genuinely hard ones to the big model. A cheap classifier can even make that routing decision. Most teams that do this cut cost and latency dramatically — and users never notice, because the hard cases still get the firepower they need.

This "cascade" pattern captures frontier quality where it matters and small-model economics everywhere else.

The takeaway

The instinct to route everything to the biggest model is expensive, slow, and usually unnecessary. Small models have quietly become capable enough to own a large share of real production work — classification, extraction, routing, rewriting, and well-scoped tasks — while being far cheaper, faster, and more private. Prototype on a strong model to prove the idea, then deliberately push the work down to the smallest model that still passes your evals, escalating only the hard cases. Do that and you will ship something that is not just cheaper, but genuinely better engineered.

Key points

• The reflex to route everything to the biggest model is often expensive, slow, and unnecessary.
• Small models now match what flagships did not long ago — the question is whether the task truly needs the big one.
• They shine at classification, extraction, routing, simple rewrites, and well-scoped, high-volume jobs.
• The advantages stack: far lower cost, faster responses, and the privacy of running on your own hardware.
• Prototype on a strong model to prove the idea, then downgrade to the smallest model that still passes your evals.
• Route by difficulty — small model for the easy majority, escalate only the hard cases.