Context windows and tokens: the unit that decides your AI feature's cost and limits
Tokens are the unit language models read, generate and bill in. Almost every practical constraint on an AI feature, cost, latency, how much context you can supply, resolves to a question about tokens.
By Quality AboveAll · · 7 min read
- A token is roughly three quarters of a word in English, and code and non-English text tokenise less efficiently.
- The context window covers input plus output together, so a long input reduces the room available for the answer.
- Filling a large window because you can usually costs money and degrades quality at the same time.
What a token actually is
Models do not read characters or words, they read tokens: fragments produced by splitting text on statistical frequency. Common words are a single token, rarer words break into pieces, and punctuation and whitespace consume tokens of their own.
The practical rule for English prose is around four characters or three quarters of a word per token. Code, structured data and languages that do not use the Latin alphabet tokenise considerably less efficiently, sometimes at several times the token count for the same information, which has real budget consequences.
The context window is a shared budget
The context window is the total the model can hold at once, covering the system prompt, conversation history, retrieved documents, the user's message and the response together. It is one budget, not separate allowances, which is why a very long input can leave insufficient room for a complete answer.
This is the cause of a common and confusing bug: responses that stop mid-sentence not because the model failed but because the remaining budget ran out. Reserving output space explicitly when assembling prompts prevents it.
Large windows are an option, not an instruction
Modern windows are large enough to hold entire documents, and the temptation is to send everything and let the model sort it out. That approach costs proportionally more on every request, adds latency, and often reduces answer quality, because relevant material gets diluted among irrelevant material.
Attention across very long inputs is also uneven, with material in the middle receiving measurably less weight than material at the beginning or end. Careful retrieval that supplies five relevant passages beats dumping fifty, which is the practical argument for investing in retrieval quality rather than window size.
A big context window is a bigger room, not a reason to fill it. What you put in still has to be worth reading.
Managing conversation history
Chat features accumulate history until they hit the window, at which point something must go. The naive approach drops the oldest turns, which works until the dropped turn contained the constraint the whole conversation depends on.
Better strategies keep the system prompt and recent turns intact while summarising older history into a compact statement of what has been established. Summarising costs a call, so trigger it on a threshold rather than every turn, and be explicit about what must never be dropped.
What this means for cost and design
Because billing is per token and output tokens usually cost several times input tokens, the two levers with the largest effect are sending less context and generating shorter responses. Trimming a bloated system prompt that ships on every single request is frequently the highest-return change available.
Caching repeated prefixes, where your provider supports it, cuts the cost of the stable portion of prompts substantially in high-volume features. We cover the full set of levers in LLM cost optimization, and model-by-model differences in choosing an LLM.
Frequently asked questions
How do I count tokens before sending a request?
Use the tokenizer library matching your model rather than estimating from character counts, particularly for code or non-English text where the ratio differs substantially from English prose.
Does a larger context window improve answers?
It permits more context, which helps when the extra material is relevant. Filling it with marginally relevant content typically reduces quality while increasing cost and latency.
Why did my response get cut off?
Usually the combined input and output exceeded the window, or the maximum output setting was reached. Reserve output space explicitly when assembling long prompts.
Costs climbing faster than usage on an AI feature? A free consultation will find where the tokens are going and which trims are safe.
