Context Window

Teams wiring a language model into a product for the first time tend to take the same path. Paste in the company document, ask a question, get an answer. On short documents it works, so the next step is a longer document.

That's where it breaks. Someone attaches a 200-page contract, asks about the termination penalty clause, and the model answers that there isn't one. The clause was on page 140. The text that actually reached the model had been cut off well before that, and everything past the cut never existed as far as the model was concerned. The worse part is that the model doesn't say it only saw the first half. It answers from what it received, in a confident sentence.

The context window is the name for that ceiling. It reframes the job: using a model isn't a matter of handing it text, it's a matter of choosing what goes into a fixed-size slot. In AI engineering it's usually the first constraint that shapes what a feature can be.

A model starts every request knowing nothing. It reads the text carried in that request, right then, and answers from that alone. The length of that text has a ceiling, measured in tokens rather than characters. A token is the smallest chunk a model splits text into — in English, roughly three quarters of a word on average.

Four things share the slot in a single request.

1. The system prompt. Fixed wording that says what this feature is and what shape the answer should take. It rides along identically on every request.
2. The conversation history. The questions and answers exchanged so far. This share keeps growing as a conversation runs.
3. Attached material. Documents pulled in by search, files the user uploaded, rows read out of a database.
4. The answer itself. Output draws on the same slot. Fill it almost entirely with input and there's no room left to write, so the response stops mid-sentence.

With a 200,000-token window, a 1,000-token prompt plus 150,000 tokens of material plus 20,000 tokens of history leaves 29,000 for the answer. If the arithmetic fits, everything looks normal. If the total goes over, one of three things happens: the request is rejected with an error, old turns are quietly dropped, or the tail of the attached material is cut off. The failures always come from the two quiet ones. An error shows up on screen; truncation leaves no signal at all, so the feature looks healthy while the answers are wrong.

It matters that the model remembers nothing between requests. A feature that appears to pick up yesterday's conversation does so because the product stored those turns and shipped them again inside today's request, not because the model retained anything. It's like handing the full packet to a stranger before every meeting — the thicker the packet, the more each meeting costs to start, every single time.

The number first appears when a team is deciding what to include and what to leave out. When the material is bigger than the window, the shape of the feature changes rather than the size of the paste.

• Long documents. Material that exceeds the window can't go in whole, so it gets split into pieces ahead of time and only the passages related to the question get attached — the retrieval-augmented generation (RAG) approach. Bigger windows don't retire this. Stuffing in unrelated text lowers the accuracy of the answer.
• Conversational features. History eventually overflows, so the team decides what to discard: drop the oldest turns, or fold the early part into a one-paragraph summary and keep only recent turns verbatim.
• Cost and latency. Billing runs on token counts, so a full window makes every answer expensive. The more text there is to read, the longer it takes for the first word to appear.
• Agent designs. When a model works through several steps with tools, each step's result piles into the next request. Around ten steps the window fills, and cutting the middle means continuing without the decisions made earlier, which is how an agent contradicts something it settled three steps ago.
• Prompt design. Choosing how many examples to include in prompt engineering is the same act of dividing the slot.

"This document won't fit in the context window as is, so we'll have to truncate."

The material is larger than the ceiling, part of it won't be delivered, and whatever isn't delivered won't appear in the answer.

"Once the conversation gets long, the early turns fall out of the window."

Old turns are being discarded, so the model no longer sees the conditions the user stated at the start.

"The window is large — should we just include every related document?"

Opening a discussion about whether the cost and the accuracy tradeoff are worth it.

At Pagedeck the phrase came up constantly while building internal document search. Theo's first version attached ten retrieved documents whole, and whenever a long one — a meeting transcript, usually — landed in the set, the documents after it got cut. From the user's side it looked like the model claimed not to know something that was visibly in the search results.

Theo changed the design to split documents by paragraph and attach only matching passages. Mia asked for the answer screen to show which document and which passage had been read, so that anything missing would be visible to the person asking rather than silently absent.

The first assumption is that a large window ends the problem. Once a million-token model exists, picking documents looks unnecessary. But fitting and using well are different things. Mixing in unrelated material makes the model reason from the wrong passage, so hallucination goes up, the bill goes up with every token included, and the wait for an answer gets longer. Designs that fill the window usually make all three worse at once.

The second is reading the window as memory. A conversation that flows makes it feel like the model retains what came before. What's actually happening is that the product resends the earlier turns on every request, and that resend is counted in tokens and billed. Leave this belief in place and there's no way to explain why long conversations get slower and more expensive, and problems caused by turns falling out of the window get blamed on model quality instead. The fix in those cases is almost never a better model; it's deciding what the product should keep resending and what it can safely summarize away.