Context Rot

Context rot is the tendency of a large language model (LLM) to produce less accurate, less consistent and less relevant output as the volume of input text it must process increases — even when the information it needs is present in that input, and even when the model's nominal context window is far from full^[1].

Background

Model vendors have competed on raw context window size, stretching from a few thousand tokens in early models to millions in current frontier systems. Window size measures capacity, not usable capacity: the amount of text a model can ingest does not guarantee that it will use that text reliably.

In July 2025, vector database company Chroma published the report Context Rot: How Increasing Input Tokens Impacts LLM Performance by Kelly Hong, Anton Troynikov and Jeff Huber, which is generally credited with coining the term^[1]. Its experiments — including simple repeated-word reconstruction, retrieval tasks and reasoning probes — found that performance degrades unevenly and often unpredictably as input length grows, and that models do not maintain consistent performance even on simple tasks as context lengthens.

Earlier research had documented related failure modes. The Stanford study Lost in the Middle (Liu et al., 2023) identified a U-shaped reliance on position: models reliably use information placed at the beginning or the end of a prompt, but are much more likely to ignore the same information placed in the middle^[2]. Summarising this line of work, Redis reports that adding roughly 20 retrieved documents (about 4,000 tokens) can pull answer accuracy down from 70–75% to 55–60%^[3].

Key factors

• Positional encoding limits — attention positions learned during training on shorter sequences extrapolate poorly to much longer inputs.
• Attention dilution — as more tokens compete for attention, the signal carried by a relevant passage is diluted by irrelevant content around it.
• Distractor sensitivity — repeated or similar-looking passages make it harder to select the correct fact, and irrelevant examples can steer generation.
• Instruction drift — in long agentic sessions, system instructions and tool contracts are followed less strictly as conversation and tool history accumulates.

Why it matters for AI

Context rot is one of the main reasons retrieval-augmented generation (RAG) remains popular despite million-token windows: the practical answer is to retrieve a small amount of relevant material rather than to stuff everything in. It also changes how agents are built — long-running agents must summarise, compact and prune their history between steps, or the second half of a task is performed on degraded context.

The economics reinforce the engineering point. Long prompts are billed per token and processed at length, so stuffing full documents into every call costs more while performing worse. Simple long-context benchmarks, which mostly test exact retrieval of planted facts, overstate real-world reliability.

>See Also

• Context Window
• Context Engineering
• Retrieval-Augmented Generation
• Prompt Engineering