How to choose a language model for your product without guessing
Public benchmarks tell you how models perform on someone else's problems. The only comparison that predicts your outcome is one built from your own tasks, and it takes about a day to assemble.
By Quality AboveAll · · 8 min read
- Build a 50 to 100 example evaluation set from your real tasks before comparing anything. Leaderboards do not transfer.
- Cost is driven by tokens per request and request volume, not by the headline price per million tokens.
- Design behind a provider-agnostic interface, because the best model for your task will change within months.
Start with the task, not the model
Write down exactly what the model must do, with real inputs and the outputs you would accept. Summarising support threads, extracting fields from invoices and drafting replies are three different tasks with different winners, and a model that leads on one may be mediocre at another.
Assemble fifty to a hundred representative examples, deliberately including the awkward ones: the ambiguous input, the very long document, the request that should be refused. This set becomes the thing you re-run on every model change for the life of the product, so it is worth an afternoon of care.
What to measure
Accuracy on your task comes first, scored consistently, whether by exact match, structured comparison, or human review of a sample. Then latency at the percentile your interface actually needs, because a model that is superb but takes eight seconds is unusable in an interactive flow and fine in a nightly batch.
Then measure instruction adherence and refusal behaviour. A model that ignores your output schema one time in twenty creates parsing failures downstream, and one that refuses reasonable requests generates support tickets. Both are easy to miss in casual testing and painful in production.
A model that is right ninety-five percent of the time and structurally parseable one hundred percent of the time is worth more than the reverse.
Understanding what you will actually pay
Headline pricing per million tokens is close to meaningless without your token profile. A retrieval-heavy feature that stuffs eight passages into every prompt can cost many times more per request than a short classification call on the same model, and output tokens usually cost several times more than input.
Estimate tokens per request from real examples, multiply by expected volume, and only then compare. Caching, prompt trimming and routing simple requests to a cheaper model routinely cut bills substantially, which we cover in LLM cost optimization.
Hosted, open weight, or self-hosted
Hosted APIs win on time to market and remove operational burden, which is why almost every product should start there. Open-weight models running on your own infrastructure win when data residency is contractual, when volume is high enough that per-token pricing exceeds the cost of running hardware, or when you need a guarantee that the model will not change under you.
That last point is underrated. Hosted models are updated, deprecated and retired on the provider's schedule, and a silent quality shift in a model your product depends on is a real operational risk. Keeping your evaluation set runnable on demand is how you detect it. Smaller options are covered in small language models versus LLMs.
Keep the decision reversible
Put every model call behind your own interface that takes a task and returns a result, rather than scattering provider-specific calls through the codebase. This costs almost nothing on day one and means switching provider is a contained change rather than an archaeology project.
Then re-run your evaluation set quarterly and whenever a provider announces something significant. The pace of change means today's correct choice has a shelf life, and teams that made switching cheap are the ones that benefit from it. It is the same API-first principle applied to an unusually fast-moving dependency.
Frequently asked questions
Should I just use the most capable model available?
Only if your evaluation shows you need it. Many production tasks are handled equally well by cheaper, faster models, and the savings compound at volume while the quality difference is invisible to users.
How often should I re-evaluate my model choice?
Quarterly is a reasonable baseline, plus any time your provider ships a significant release or you notice a change in output quality. Keeping the evaluation set ready makes this cheap.
Is an open-source model good enough for production?
For many well-defined tasks, yes. The deciding factors are usually operational rather than qualitative: whether you want to run the infrastructure, and whether data residency or model stability requirements justify it.
Choosing a model for something you are about to build? A free consultation gets you an evaluation plan built around your task rather than a leaderboard.
