LLM cost optimization: the levers that cut spend without users noticing

Find out where the money goes

Attribute spend per feature and per request type before changing anything. A single expensive feature usually dominates, and optimising the cheap ones is effort spent for nothing.

Break each request into its components: system prompt, retrieved context, conversation history, user input and output. That breakdown almost always reveals something surprising, most often that retrieved context is several times larger than anyone intended.

Trim the prompt that ships every time

System prompts accumulate instructions and rarely lose them. Since they are sent on every single request, a bloated system prompt is a fixed tax on your entire volume, and reviewing it against your evaluation set frequently removes a substantial fraction with no measurable quality change.

The same applies to retrieved context. Sending eight passages when three would do triples that portion of the bill, and often reduces answer quality by dilution, as covered in context windows and tokens. Better retrieval is a cost optimisation as well as a quality one.

Every sentence in your system prompt is charged once per request, forever. Read it like an invoice.

Cache aggressively

Exact-match caching of identical requests is trivial and effective wherever repeated questions occur, which is most support and documentation use cases. Semantic caching, matching near-identical questions, extends the hit rate further with a small relevance risk worth measuring.

Provider-side prompt caching for stable prefixes, where available, reduces the cost of the unchanging portion substantially in high-volume features. Structuring prompts so the stable part comes first is a small change that makes this possible.

Route by difficulty, and cap the output

Not every request needs your most capable model. Classify and route: a small model handles the routine majority, escalating only what needs more, as covered in small language models versus LLMs. Given typically skewed request distributions, the savings are large.

Output tokens usually cost several times input tokens, so instructing concise responses and setting maximum output length cuts cost directly, and usually improves the user experience at the same time. Verbose AI output is rarely what anyone wanted.

Batch what can wait, and cap what cannot

Anything not needed interactively should use batch processing where your provider offers it at a discount. Overnight classification, enrichment and summarisation are natural candidates and often represent a large share of total volume.

Finally, enforce hard limits: per-user and per-session token budgets, global spend alerts, and circuit breakers. An unbounded retry loop against a paid API can generate a serious bill overnight, and the control that prevents it is a few lines of code. Monitoring is covered in AI observability.

Frequently asked questions

What is the biggest single saving usually available?

Trimming what is sent on every request: an accumulated system prompt and over-generous retrieval. Both are common and neither costs quality to fix.

Does caching hurt answer freshness?

Exact-match caching with sensible invalidation on content change is safe. Semantic caching needs a similarity threshold you have actually measured, or it will serve subtly wrong answers.

How do we prevent a runaway bill?

Per-user and per-session token budgets, spend alerting with thresholds, and circuit breakers on error loops. Put them in before launch, not after the first incident.

AI spend rising faster than usage? A free 30-minute consultation will profile where the tokens go and which cuts are safe.