Context Windows and Token Cost

Updated Nov 16, 2026 · 5 min read
On this page

    A chat assistant works well in a demo. In production, long conversations start forgetting what the user said at the beginning, some requests fail with "context length exceeded", and the cost per conversation keeps climbing the longer people chat. All three come from one fact: the model is stateless, and everything it needs must fit in one context window, paid for in tokens, on every call.

    How it works

    The window is everything the model sees

    The context window is the maximum number of tokens the model can handle in one call: the input you send plus the output it writes. Nothing outside it exists for the model. There is no hidden memory between calls; a chat "remembers" only because the app resends the conversation every time.

    Every call packs system prompt, history, retrieved text and the new message into one budget, and must leave room for the answer.
    Every call packs system prompt, history, retrieved text and the new message into one budget, and must leave room for the answer.

    A typical request contains:

    • the system prompt (instructions, persona, tool definitions);
    • the conversation history;
    • retrieved context (documents from RAG, tool results);
    • the user's new message;
    • and the answer the model will write, which counts against the same window.

    Every turn pays for the whole history again

    Because the app resends the history, input tokens per call grow with each turn, and the total over a conversation grows roughly with the square of its length.

    Example numbers: a 1,000-token system prompt plus 500 new tokens per turn. Six turns cost 16,500 input tokens, not 9,000.
    Example numbers: a 1,000-token system prompt plus 500 new tokens per turn. Six turns cost 16,500 input tokens, not 9,000.

    Pricing basics

    API pricing is per token, with separate prices for input and output; output usually costs several times more per token. Many providers also discount cached input: when the start of a prompt is identical to a recent request, that prefix is cheaper and faster (see Prefix Caching). Self-hosted models follow the same logic in GPU time: long prompts take longer to process, and every output token is one more decoding step.

    Bigger windows are not free memory

    Windows have grown from a few thousand tokens to hundreds of thousands, and some models go beyond that. Two limits remain even when the text fits:

    • Cost and latency grow with every token you send. Processing a very long prompt can take seconds before the first output token.
    • Attention quality varies with position. Many models use information at the start and end of a long prompt more reliably than information buried in the middle. Putting everything in "just in case" can make answers worse, not better.

    Trade-offs and costs

    ApproachGoodBad
    Send everythingSimple, no retrieval to buildSlow, expensive, quality drops on long inputs
    Sliding window (last N turns)Cheap, predictableForgets early facts
    Running summaryKeeps the gist of long chatsLoses detail; the summary can be wrong
    Retrieval (store and fetch relevant pieces)Small prompts, precise contextMore components, retrieval can miss
    Real products usually mix all three: recent turns verbatim, an older summary, and retrieval for specific facts.
    Real products usually mix all three: recent turns verbatim, an older summary, and retrieval for specific facts.

    In production

    • Count before you call. Use the model's tokenizer to measure the prompt, and trim lower-priority parts (oldest history first, then the least relevant documents) until there is room for the answer.
    • Set max output tokens on every call. It bounds cost and latency and reserves space correctly.
    • Keep the stable part first. System prompt and tool definitions at the start, unchanged between calls, so prefix caching can work.
    • Monitor tokens per request (p50, p95) and per conversation; long tails often hold most of the cost.
    • Failure modes: "context length exceeded" errors from one oversized document; answers ignoring a fact placed mid-prompt; costs spiking because a tool returns a huge JSON blob into the context.

    When to reach for it

    Every LLM feature needs a context budget, but it matters most for multi-turn chat, RAG with many documents, agents whose tool outputs pile up in the context, and anything that processes long files.

    Example

    Assume a support assistant with a 1,000-token system prompt, 500 new tokens per turn and an average of 10 turns per conversation, at an example price of $2 per million input tokens.

    • Input tokens per conversation: the call for turn t sends 1,000 + 500·t tokens, so the total is 10 × 1,000 + 500 × (1 + 2 + … + 10) = 10,000 + 27,500 = 37,500 tokens.
    • Input cost per conversation: 37,500 × $2 / 1M ≈ $0.075. At 1 million conversations a month, about $75,000 for input alone.

    What the numbers mean: history dominates as chats get longer. Keeping only the last 4 turns verbatim plus a 300-token summary caps each call at roughly 1,000 + 300 + 4 × 500 + 500 ≈ 3,800 tokens, and later turns get cheaper rather than more expensive.

    Common mistakes

    • Assuming the model remembers earlier calls. It doesn't; the app resends context.
    • Forgetting the output when checking the window.
    • Putting a changing value (like the timestamp) at the very start of the prompt, which breaks prefix caching.
    • Dumping whole documents or raw tool output into the prompt instead of the relevant part.
    • Estimating cost from one turn instead of whole conversations.

    Say this in the interview

    "The model is stateless, so each call must carry everything: system prompt, history, retrieved text and the new message, plus room for the answer, inside the context window. Because history is resent, cost per conversation grows faster than linearly. I'd count tokens before each call, cap output, keep a stable prefix for caching, and manage long chats with recent turns plus a summary plus retrieval, rather than relying on a huge window."

    Interview questions on this

    1. The model has a 1M-token window. Why not send everything? Cost and latency grow with tokens, and quality drops for information buried in long prompts.
    2. How does a chatbot "remember" a conversation? The app stores the messages and resends them (or a summary) on every call.
    3. How do you stop a long agent run from overflowing the window? Summarise or drop old tool outputs, store large results outside the prompt and pass references, and cap tool output size.
    4. Why is output priced higher than input? Each output token needs its own sequential decoding step; input tokens are processed in parallel.
    5. What breaks prefix caching? Anything that changes the beginning of the prompt between calls, such as timestamps, user IDs or reordered tools.

    Used in

    • AI Engineering: Tokens and Tokenization; Conversation Memory for LLM Apps; Prefix Caching; Semantic Caching; How Would You Cut LLM Cost by 10x?
    • AI System Design: Tokens, context window and cost; Memory; Design ChatGPT.