Context Rot: Why a Bigger Context Window Doesn't Fix Your Agent

When an agent loses the plot or returns a wrong answer, the reflex is to give it more context. Add the missing files, move to a model with a bigger window, paste in more of the document. The evidence says that reflex makes things worse.

The window is not a container you fill. It is a surface the model has to look across, and the failure is structural rather than cosmetic. Context rot is the name for it, it is reproducible, and it starts earlier than most teams expect.

What is context rot?

Context rot is the decline in answer accuracy a language model shows as its input context grows longer. Chroma's Context Rot study tested 18 frontier models and found accuracy degraded non-uniformly in every one of them, beginning well before the advertised context limit. It is a property of the models, not a mistake in your prompt.

The study, by Kelly Hong, Anton Troynikov, and Jeff Huber, extended needle-in-a-haystack testing to measure reasoning accuracy as context grew, across models including GPT-4.1, the Claude 4 family, Gemini 2.5, and Qwen3. They varied needle-question similarity, the number of distractor documents, needle-haystack similarity, and haystack structure. No model held its short-context accuracy as input length increased, and the decline was irregular rather than a clean cliff at the limit. The full report is on Chroma's research page.

Why doesn't a bigger context window fix it?

Because the degradation is not caused by hitting a token ceiling. Models lose accuracy progressively as input grows, so a larger window offers more room to accumulate the same decay. The advertised limit is a capacity ceiling, not a quality floor.

The container mental model is the error. A context window is not a bucket where everything you add stays equally usable. Every token competes for the model's bounded attention, so adding context changes how the model weights what is already there. A team that "fixes" a failure by adding more context multiplies the competition rather than removing the cause. What you retrieve into the window matters as much as how large the window is, which is why RAG, MCP, and direct API tooling are not interchangeable.

What does the lost-in-the-middle U-curve show?

That position changes the answer. The lost-in-the-middle finding, from Liu and colleagues, shows language models use information best when it appears at the beginning or end of the context and worst when it sits in the middle. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang measured this in multi-document question answering and in key-value retrieval, and accuracy followed a U-shape by position across the models they tested.

In their 20-document condition, the strongest models held roughly 70 to 75% accuracy when the answer sat at the start or the end, and fell to the mid-50s when the same passage sat in the middle. The paper is "Lost in the Middle: How Language Models Use Long Contexts".

Nothing about the document changed between those rows. Only where it sat.

Why does position matter more than volume?

Because attention is distributed across the whole window, so material in the middle competes with everything around it for a limited share of focus. A transformer computes attention over the entire input, and that attention is not evenly shared. Content in the middle is surrounded on both sides, so it gives up the positional advantage an opening or closing position grants.

Chroma's study reinforced this from the other direction: adding distractor documents, even plausible-looking ones, pushed accuracy down, and the distractors that most resemble the query do the most damage. That is why volume is a cost rather than a benefit. A larger window does not create extra focus. It divides the same focus across more tokens.

How do you keep context from rotting?

Keep the window small on purpose. Retrieve fewer and better documents instead of filling the budget, place the instruction and the decisive fact near the start, and load context in layers rather than dumping it. These are working practices rather than measured constants, but each one counters a specific failure above.

RAG pipelines usually over-fetch, stuffing the window with every loosely relevant document to lift recall. The lost-in-the-middle result says that is the wrong trade, because fewer higher-precision documents keep signal density high. Progressive disclosure reveals material only when a step needs it, and persisting durable state across turns stops the model re-accumulating the same context every session. The same logic caps how many tools you should expose at once: every description is more tokens competing for the same attention.

What should you put at the edges of the window?

Put the instruction, the system prompt, and the single decisive fact at the start, and the latest action-relevant state at the end. Reserve the middle for supporting detail you can afford to lose. Position is a design decision you control even when you cannot shrink the total.

Concretely, open with a short unambiguous instruction and place the fact the whole answer depends on immediately after it, then let the closing tokens carry the most current state. Mid-window is for appendices, logs, and background you would not mind the model skipping. When a task needs data that cannot live in the window at all, move it behind a query so the model pulls only what each step requires. Askpert is one option for that pattern, keeping the underlying material server-side and returning only the answer, though the principle applies to any retrieval layer you build yourself.