Context Rot

Context rot is the gradual loss of reliability that occurs when an AI model receives more context than it can use effectively. Even when an advertised context window can technically hold hundreds of thousands of tokens, a large language model may overlook relevant facts, follow outdated instructions, or reason less accurately as the input grows. A 2025 Chroma context rot study observed this nonuniform performance across 18 models and several controlled tasks.

How Context Rot Happens#

Long inputs place greater demands on the model's attention mechanism. Important evidence must compete with repeated instructions, irrelevant documents, tool outputs, and older conversation turns. Context position, semantic similarity, conflicting facts, and task complexity can all affect what the model uses.

The 2024 RULER long-context benchmark found that models performing well on simple retrieval often declined as sequence length and task complexity increased. The 2025 NoLiMa benchmark revealed larger drops when finding an answer required semantic reasoning rather than matching identical words. There is therefore no universal token count—including for Gemini models—at which context rot begins; the threshold depends on the model, prompt structure, and task.

Real-World Examples#

• Customer-support assistants: A chatbot given years of tickets may prioritize an obsolete policy or miss a recent account update. Research using LongMemEval conversational memory and the multimodal LoCoMo benchmark shows that extracting, updating, and reasoning over long interaction histories remains challenging.
• Visual inspection agents: A vision-language model monitoring a factory can become less dependable if every frame, detection, and maintenance log is placed into one prompt. A better workflow uses Ultralytics YOLO26 to extract concise visual facts before language-model reasoning.
• Coding agents: Loading an entire repository, every tool definition, and complete terminal history can obscure the current objective. Anthropic’s context-engineering guidance recommends curating context, while its Agent Skills approach loads detailed resources only when needed.

This YOLO predict workflow demonstrates converting raw detections into compact, structured context:

from ultralytics import YOLO

model = YOLO("yolo26n.pt")
result = model("https://ultralytics.com/images/bus.jpg")[0]
labels = [model.names[int(cls)] for cls in result.boxes.cls]
context = {"frame": 1, "objects": sorted(set(labels))}
print(context)

How To Reduce Context Rot#

• Retrieve only relevant evidence: Use semantic chunking and a reranker instead of sending every available document.
• Compress older information: Replace long histories with verified summaries, decisions, and unresolved tasks. Research suggests that input length can reduce performance even after successful retrieval.
• Preserve stable prefixes: OpenAI prompt caching and Gemini context caching can lower repeated-processing costs, although caching alone does not improve context quality.
• Use sliding windows: Google recommends context-window compression for long live sessions, retaining recent information while evicting older tokens.
• Evaluate at realistic lengths: Apply model monitoring and reproduce controlled tests with the open context rot toolkit.

Context rot differs from hallucination, which is an unsupported output; catastrophic forgetting, which changes model knowledge during training; and data drift, which reflects changing production inputs. Context rot is primarily an inference-time failure of context selection and reasoning, making effective context engineering its main defense.