Context Windows, Explained

The model's working memory

Every model spec proudly lists a number: an 8k context window, 128k, 200k, a million. It sounds like a feature you should simply want more of. To use models well, though, you need to understand what the number actually measures, why bigger is not automatically better, and how to treat it as a resource to manage rather than a bucket to fill.

A context window is the maximum amount of text a model can consider at once, measured in tokens. Crucially, it covers everything in a single turn: your instructions, the entire conversation so far, any documents you attach, and the model's own answer as it is generated. Think of it as the model's working memory — its desk. Everything it can "see" while answering has to fit on that desk at the same time.

First, what is a token?

Since the window is measured in tokens, the unit matters. A token is a chunk of text — very roughly three-quarters of a word in English. "Cat" is one token; a longer or unusual word might split into several. A useful rule of thumb: 100 tokens is about 75 words, so a 200,000-token window holds roughly 150,000 words — a couple of full-length books at once.

That sounds like more than anyone could need. So why isn't the answer just "always pick the biggest window"?

Bigger isn't a free lunch

Large context windows are genuinely useful — feeding in a long report, a whole codebase, or a lengthy conversation. But filling them carries three real costs that the headline number hides.

• You pay for every token. Most APIs bill by tokens processed. Stuffing 100,000 tokens of context into every request, when 2,000 would do, can multiply your bill by fifty for no benefit.
• Latency rises. More context means more for the model to process before it answers. Bloated prompts feel sluggish, and at scale that lag compounds.
• Quality can actually drop. This is the counterintuitive one. Research on "lost in the middle" shows that models pay the most attention to the beginning and end of a long context and can genuinely overlook details buried in the middle. Bury the key fact in the center of 150,000 tokens and the model may answer as if it were never there.

A full context window is not a well-informed model — it is often a distracted one. Relevance beats volume almost every time.

Context rot and long conversations

There is a practical version of this that everyone has felt: the long chat that slowly goes off the rails. As a conversation grows, two things happen. Eventually it exceeds the window, and the oldest messages silently fall out of view — which is why an assistant can "forget" what you told it an hour ago. Even before that limit, a long, meandering history dilutes the important parts with noise, and answers drift. Practitioners sometimes call this context rot.

The fix is often simple: start a fresh conversation for a new task rather than letting one thread sprawl forever.

Treat the window as a budget

The mental shift that makes you good at this: stop trying to fill the context and start trying to spend it wisely. Send the least context that lets the model answer well.

• Retrieve, do not dump. Instead of pasting an entire 300-page manual, use retrieval (RAG) to pull in only the few passages relevant to the question. Better answers, lower cost.
• Summarize long histories. Rather than carrying every message verbatim, periodically compress the conversation into a short summary and continue from that.
• Put the important things at the edges. Place the most critical instructions and facts near the start or end of the prompt, where attention is strongest — not buried in the middle.
• Start fresh when the task changes. A clean window beats a cluttered one. Do not make the model wade through irrelevant history.

A common mistake

A pattern shows up constantly in teams new to building with LLMs: when answers are not good enough, they reflexively stuff more into the context. The entire knowledge base, the full conversation history, every document that might be relevant — all crammed in, on the theory that more information can only help.

It usually hurts. The key fact gets buried in the middle where attention is weakest, the bill balloons, responses slow down, and quality often drops rather than rises. The model is not better informed; it is overwhelmed.

The fix is the opposite instinct: when results disappoint, try sending less but more relevant context, not more. Retrieve the handful of passages that actually bear on the question and let the model focus. Curating beats cramming almost every time.

The takeaway

The context window is the model's working memory, measured in tokens, and it has to hold your prompt, the conversation, your documents, and the answer all at once. A bigger window is a useful capability, not a free upgrade: it costs money and latency, and a stuffed window can actually degrade quality as key details get lost in the middle. The skilled move is to treat context as a budget — feed the model the least, most relevant information it needs, and spend every token on something that earns its place.

Key points

• A context window is the model's working memory, measured in tokens, holding your prompt, the conversation, documents, and the answer.
• Roughly 100 tokens is 75 words, so a 200k window holds about two books at once.
• Bigger is not automatically better: you pay per token, latency rises, and quality can drop as details get "lost in the middle."
• Long conversations eventually push old messages out of view, which is why assistants seem to "forget."
• Treat context as a budget — retrieve only relevant chunks, summarize long histories, and put key facts at the edges.
• Start a fresh chat when the task changes; a clean window beats a cluttered one.