Context Window

A context window is the maximum number of tokens a model can take into account in a single request. For a large language model (LLM), everything it reads and writes shares that budget: the system prompt, tool definitions, conversation history, retrieved documents, the user's question, and the reply it generates. Text outside the window does not reach the model at all, which is why a chatbot loses track of the start of a very long conversation once earlier turns are cut. Windows have grown from 2,048 tokens in GPT-3 to 128K tokens in GPT-4 Turbo in late 2023 and 1 million in Google's Gemini 1.5 Pro in 2024, and Meta's Llama 4 Scout advertises 10 million.

What Fills the Context Window#

Tokenization splits text into subword pieces that average about four characters of English, so a 128K-token window holds the equivalent of roughly 300 pages. A vision-language model (VLM) also spends tokens on images, usually hundreds of patch tokens per image, so a long video fills the window far faster than its transcript would. Generated tokens count against the same limit, and providers set a separate cap on output length inside it. Applications that run long conversations or AI agents therefore assemble the prompt on every turn and decide what to keep:

Why Windows Are Limited#

Two costs grow with length. Self-attention compares every token with every other token, so attention compute grows quadratically with the number of tokens. During text generation, the KV cache keeps the keys and values of every earlier token: Llama 3.1 8B, with 32 layers and 8 key-value heads of 128 dimensions each, stores 128 KiB per token in 16-bit precision, or 16 GiB at its full 131,072-token window, more than the 16 GB its weights occupy.

The window is also set by training. A model trained on 8K-token sequences has never seen the positions beyond them, so extending it means rescaling its rotary position embeddings (RoPE) with methods such as YaRN and fine-tuning briefly on longer text. Serving long windows relies on memory-saving techniques: flash attention computes exact attention without storing the full score matrix, grouped-query attention (GQA) shrinks the KV cache, sliding window attention limits each token to recent neighbors, and ring attention splits one sequence across many GPUs.

Advertised vs. Effective Context#

A model that accepts a million tokens does not use all of them equally well. The 2023 study Lost in the Middle found that models answer best when the relevant passage sits at the start or end of the prompt and worst when it sits in the middle. Needle in a haystack (NIAH) tests measure recall of one inserted fact at every depth and length, and the harder RULER benchmark found that only half of 17 models claiming 32K tokens or more performed satisfactorily at 32K. The gradual decline as inputs grow is called context rot, and retrieval-augmented generation (RAG) and context engineering counter it by putting only the relevant material in the window.

Context Window vs. Related Terms#

• Receptive field: The region of an input image that influences one output of a convolutional neural network (CNN). It is fixed by kernel sizes and depth, while a context window is a token budget.
• Batch size: The number of independent sequences processed together. The context window bounds the length of each one.
• Memory: Agent memory, such as episodic memory stored in a vector database, persists outside the window and is copied back into it when relevant.

Context Windows in Open-Vocabulary Detection#

Text encoders inside vision models have windows too. Ultralytics YOLOE and YOLO-World encode each class prompt with a CLIP-family text encoder whose context window is 77 tokens, and the Ultralytics text encoder truncates longer prompts by default. The same CLIP tokenizer shows what gets cut:

import clip
from clip.simple_tokenizer import SimpleTokenizer

prompt = (
    "a warehouse worker wearing a yellow high-visibility vest, a white hard hat, blue gloves and clear safety "
    "glasses, standing beside a red forklift that carries wooden pallets stacked with brown cardboard boxes, "
    "under bright ceiling lights in a narrow aisle between tall metal shelves, next to a fire extinguisher "
    "mounted on the wall, a green emergency exit sign, an open loading dock door and a black dog"
)

# CLIP's BPE tokenizer wraps the text in start and end tokens
tokenizer = SimpleTokenizer()
print(f"Prompt length: {len(tokenizer.encode(prompt)) + 2} tokens")

# The text encoder reads a fixed 77-token window, so truncation drops the tail
tokens = clip.tokenize(prompt, truncate=True)[0]
kept = tokenizer.decode(tokens[1:-1].tolist())
print(f"Window: {len(tokens)} tokens ending in '{' '.join(kept.split()[-4:])}'")
print(f"'black dog' kept: {'black dog' in kept}")

The prompt is 80 tokens long, so the window keeps the first 75 text tokens plus the start and end markers, ends at "loading dock door and", and drops "a black dog" before the encoder ever sees it.