Small language models versus large ones: when the smaller model is the better engineering choice
The reflex to use the largest available model on every task is expensive and often unnecessary. For narrow, well-defined jobs, a small model is frequently faster, cheaper and entirely good enough.
By Quality AboveAll · · 8 min read
- Small models match large ones on narrow, well-specified tasks like classification, extraction and routing.
- They enable on-device and on-premise deployment, which solves data residency and latency problems outright.
- Routing simple requests to a small model and hard ones to a large model captures most of the savings with little quality loss.
What counts as small, and why it matters
Small language models are those compact enough to run on modest hardware, a single commodity GPU, a server CPU, or increasingly a phone or laptop. The interesting shift is not that they exist but that the recent generation performs well enough on constrained tasks to be genuinely usable in products.
The advantages compound: lower cost per call, lower latency, and the option to run where your data already lives rather than shipping it to a third party. For some organisations that last point is not an optimisation but a precondition for shipping at all.
Where small models are genuinely sufficient
Classification, routing, extraction from consistent formats, sentiment and intent detection, and short-form rewriting are all tasks where a well-chosen small model, given a clear prompt and good examples, performs comparably to something far larger. These also happen to be the highest-volume tasks in most products.
Fine-tuning shifts the balance further. A small model tuned on a few thousand examples of one specific task routinely outperforms a much larger general model on that task, at a fraction of the running cost. This is one of the clearest remaining cases for fine-tuning.
Nobody needs a frontier model to decide whether a support ticket is about billing or delivery.
Where you still want the large model
Open-ended reasoning across several steps, synthesis over long and messy context, nuanced writing, and anything requiring broad world knowledge remain clearly better on large models. So does robustness to unusual phrasing, which matters when real users type things nobody anticipated.
Complex tool use is another. Agents that plan sequences and recover from failures need the reasoning depth larger models provide, which is why our agent work generally puts the large model at the planning layer even when smaller models handle individual steps.
Routing: the pattern that captures both
Rather than choosing once, classify each request and route it. A small, fast model handles the common simple cases; anything it flags as ambiguous or complex escalates to the larger model. Because request distributions are usually heavily skewed toward the simple, the savings are substantial.
The engineering requirement is a reliable escalation signal and a way to measure quality on both paths, so the router does not quietly degrade the experience for the cases it keeps. That means the evaluation discipline in our LLM evaluation guide applied per path, not in aggregate.
On-device and on-premise deployment
Running a model on the user's device removes network latency, works offline, and means the data never leaves the device, which resolves an entire category of privacy questions rather than mitigating them. The constraints are memory, battery and a meaningfully harder update story.
On-premise deployment addresses the same concerns at organisation scale, and for regulated clients it is frequently the only acceptable architecture. We cover the mobile trade-offs in AI in mobile apps and the data handling questions in data privacy in AI systems.
Frequently asked questions
Are small models much worse than large ones?
On broad reasoning and open-ended generation, yes. On narrow well-defined tasks the gap is often negligible, and a fine-tuned small model can beat a large general one on its specific task.
Can a small model run on a phone?
Modern small models run on current flagship devices, with memory and battery as the practical limits. Whether it is worth it depends on whether offline operation or on-device privacy is a real requirement.
How do I decide which tasks to route where?
Evaluate both models on your golden dataset per task type. Route the tasks where the small model matches, escalate the rest, and monitor quality on each path separately.
Paying frontier prices for classification work? A free consultation will show you which of your calls could route to something smaller without users noticing.
