Context Rot: Why AI Models Lose Track of Long Prompts

Every new AI model seems to come with a huge context window: 128K, 200K, even a million tokens. Yet anyone who uses these models often has seen the same problem. In a long chat, the model starts to ignore instructions from earlier. Give it ten documents, and its answer may only use two of them.

Both things can be true. A model may be able to accept a huge amount of text without being able to use all of it well.

Researchers have measured this gap. They have also found some surprising reasons for it. This problem is often called "context rot." In this post, we will look at what we know about it, why models often lose information in the middle of a prompt, and why common benchmark results do not tell the whole story.

Try it yourself

You can see the basic problem in five minutes, without writing any code.

Take a long document, perhaps 30 or 40 pages. Add one clear sentence that does not appear anywhere else, such as: "The migration deadline was moved to March 12."

Now make three copies of the document. Put the sentence near the start of the first copy, in the middle of the second, and near the end of the third. Open a fresh chat for each copy and ask the same question: "When was the migration deadline moved to?"

The model will usually do well when the answer is near the start or the end. The middle is less reliable, especially when the text around the answer is about a similar topic. That last detail will matter later.

This is more than a neat demo. In the 2023 paper that gave the effect its name, Liu and other researchers ran a similar test at scale. They tested GPT-3.5, Claude, and several open models. The answer was hidden among 10 to 30 documents.

The results formed a U-shaped curve. Models did better when the answer was near the start or the end, and worse when it was in the middle.

With 20 documents, GPT-3.5-Turbo was correct 75.8% of the time when the answer came first and 63.2% when it came last. Accuracy fell to about 54% in the middle. The same model scored 56.1% when it had no documents and had to answer from memory.

In other words, when the answer was present but buried in the middle, the model did worse than when it had no documents at all.

Position clearly matters. But why? One process gives the start of a prompt an advantage. Another gives the end an advantage. The middle is where neither one helps much.

Attention is a fixed pie

First, it helps to understand what happens as a prompt gets longer.

When a transformer processes text, each token calculates how relevant the other tokens are. These relevance scores go through a function called softmax, which makes them add up to 1. You can think of attention as a fixed pie.

In a prompt with 1,000 tokens, an important sentence may get a large slice. In a prompt with 100,000 tokens, that same sentence has to compete with 99,999 other tokens. Its slice can shrink even though the sentence itself has not changed.

The signal does not suddenly disappear. It slowly gets lost in the noise. That is why performance tends to decline gradually instead of falling off a cliff. The model does not simply "run out of room." It becomes harder for the model to pick out one useful piece of information from a growing crowd.

Chroma's 2025 report, "Context Rot," tested 18 models, including Claude Opus 4, GPT-4.1, and Gemini 2.5 Pro. It found that length alone was not the biggest problem. Distracting information did more damage.

Text that looks relevant but does not answer the question can make retrieval harder than completely unrelated filler. Even one distractor caused a clear drop in performance, and four distractors made it worse.

So the problem is not just "too much text." It is often "too much text that looks useful." If the answer in your test was surrounded by similar material, this is probably part of what made it hard to find.

Why the start gets special treatment

Now we come to the strange part: why do models often remember the start of a prompt so well?

In 2023, researchers at MIT and Meta were trying to help models process an endless stream of text while using a fixed amount of memory. A simple approach was to keep only the latest few thousand tokens and remove the oldest ones. The oldest tokens seemed like they should matter the least.

But removing the first tokens caused a huge failure. For Llama-2-13B, perplexity, a rough measure of the model's confusion, jumped from 5.40 to 5,158. The model did not just get a little worse. It fell apart.

When the researchers kept only the first four tokens along with the recent text, performance returned to normal.

They called the reason "attention sinks." Softmax has to place all of its attention somewhere at every layer and every step, even when none of the earlier text is useful. Models learn to use the first few positions as a place to put unused attention.

After the earliest layers, many attention heads focus heavily on the first token. In deeper layers, some put more than half of their attention there. The start of the prompt becomes a kind of parking place for attention.

The words in those first positions do not seem to matter much. The researchers replaced the original first tokens with newline characters. Perplexity was still 5.60, close to the original score of 5.40. The position mattered far more than the content.

So models do not remember the start only because they decide it is important. The first few positions sit in a special place that attention heads return to again and again.

The end of the prompt gets an advantage for a different reason. Most modern models use a positional system called RoPE. With this system, attention tends to weaken as the distance between tokens grows. Recent text stays clear, while distant text becomes harder to use.

There is another issue too. Many models that advertise context windows of 128K tokens or more were first trained on much shorter sequences. Their context windows were later extended. This lets them accept longer inputs, but their view of distant positions can be less precise.

The result is the U-shape. The start benefits from attention sinks. The end benefits from being recent and close. The middle gets neither advantage.

There is also a simpler possible explanation. In real writing, important information often appears at the edges: abstracts, introductions, conclusions, and the latest messages in a conversation. Models may have learned that the middle is often less useful.

That idea makes sense, but it has not been proven in the same way. Attention sinks and the effect of distance have been measured directly. The idea that training taught models to treat middles as filler is still a reasonable guess.

Most benchmarks use easy mode

This brings us to the large context numbers shown on model cards.

A common long-context test is called "needle in a haystack." A sentence is hidden inside a long piece of text, and the model is asked to repeat it. Model launches often show the results as a green chart, with success across many positions and input lengths. Today's leading models really are very good at this test.

But exact retrieval is one of the easiest long-context tasks. The question and the hidden sentence often use the same words. The model can match those words instead of understanding how the information connects. It is close to keyword search dressed up as a reasoning test.

The 2025 NoLiMa benchmark removed this shortcut. Its main rule was simple: the question and the hidden fact should share as few words as possible.

Here is one of its examples. The hidden sentence says: "Actually, Yuki lives next to the Semper Opera House." The question asks: "Which character has been to Dresden?"

There is no obvious phrase to match. The model must know that the Semper Opera House is in Dresden, then connect that fact to Yuki. This is much closer to how people need models to use context in real work.

Performance dropped sharply. NoLiMa tested 13 models, each claiming a context window of at least 128K tokens. All of them did very well on short inputs. But at only 32K tokens, 11 of the 13 models fell below half of their own short-context score. GPT-4o was among the strongest models and still dropped from 99.3% to 69.7%.

NVIDIA's RULER benchmark reached a similar conclusion in a different way. RULER defines a model's effective context length as the longest input where the model still stays above a set quality level. It then compares that length with the advertised context window.

GPT-4 advertised 128K tokens but had an effective length of 64K in this test. Llama 3.1 8B advertised 128K but had an effective length of 32K. Yi-34B advertised 200K but also had an effective length of 32K. One model advertised a million tokens but had an effective length below 4K.

The advertised context window tells you how much text a model can accept. The effective context length tells you how much of that text it can use well. Only the first number usually appears on the box.

This matters to me personally. My own benchmark includes a long-context section, and recent models often score a perfect 1.00 on it. I do not take that to mean that long context is solved. A perfect score on a retrieval task shows that the models passed an easier test at the lengths I used. It does not show that they would handle a NoLiMa-style task at ten times the length.

Understanding what kind of test you are running matters more than the score alone.

Chroma found something else that is even more surprising. Across all 18 models in its study, models were better at finding information in shuffled text than in clear, well-organized writing.

When researchers broke a document into sentences and mixed up their order, the hidden fact became easier to find. Clear writing may create a story or flow that the model follows. A fact that does not fit that flow may then be easier to overlook.

We do not yet know the full reason, but the result challenges the idea that models read long documents the way people do.

What you can do

The research points to a few practical habits:

• Put instructions at the start, then repeat the request at the end. These are the two strongest positions. In a long prompt, ask the main question again after the documents.
• Keep the middle lean. Do not include ten documents when three are enough. Every unnecessary token competes with useful information. Related but incorrect material can be especially distracting.
• Do not assume the model will use something just because it is in the prompt. If a fact is critical, pull it out and state it clearly. Do not leave it buried on page 40.
• Repeat important details in long chats. An instruction from an hour ago may now be too far away. A one-line reminder at the point where it matters is more reliable.
• Be careful with advertised context sizes. RULER and NoLiMa suggest a practical starting point: assume reliable performance at about one-quarter to one-half of the claimed length. Then test the model with your own tasks and input sizes before depending on it.

A few honest caveats

The main findings in this post have been repeated across studies. These include the U-shaped performance curve, attention sinks, the test with newline characters, the gaps found by NoLiMa and RULER, and the effects of distractors and shuffled text.

The explanations are not all equally certain. Attention sinks and the effect of distance have been measured directly. The idea that training data taught models to treat the middle as filler is a sensible theory, but I have not seen a study isolate and prove it.

This field also moves quickly. These results come from studies published between 2023 and 2025. Newer leading models handle long prompts better than many models in the older papers. Still, the basic shape of the problem has appeared in every generation tested so far. The exact numbers will change, but the problem has not disappeared.

Sources

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023. arxiv.org/abs/2307.03172
• Xiao et al., Efficient Streaming Language Models with Attention Sinks, 2023. arxiv.org/abs/2309.17453
• Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, 2025. arxiv.org/abs/2502.05167
• Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models?, 2024. arxiv.org/abs/2404.06654
• Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance, 2025. research.trychroma.com/context-rot