Context Rot: How Increasing Input Tokens Impacts LLM Performance
by Chroma
Eighteen frontier models, and every one degrades as the context grows — even on a task a child could do.
Overview
Published 14 July 2025 by Kelly Hong, Anton Troynikov and Jeff Huber, this report attacks the assumption that a long context window is processed uniformly. It runs 18 models — Claude Opus 4, Sonnet 4, Sonnet 3.7, Sonnet 3.5 and Haiku 3.5; OpenAI o3, GPT-4.1 with mini and nano, GPT-4o, GPT-4 Turbo and GPT-3.5 Turbo; Gemini 2.5 Pro, 2.5 Flash and 2.0 Flash; and Qwen3 at 235B, 32B and 8B — across eleven sections of experiments. The Needle in a Haystack extension replaces lexical matching with semantically similar needles, varies needle-question similarity, injects one and then four distractors, compares needle-haystack topical alignment, and sets a coherent haystack against a shuffled one. LongMemEval contrasts a roughly 113k-token full conversation against a roughly 300-token focused extract containing the same relevant content. The repeated-words task asks a model to replicate a sequence containing one unique word, across 1,090 context-length variations from 25 to 10,000 words. The findings: performance declines with input length across every model tested; lower needle-question similarity degrades faster; distractors compound, with Claude models tending to abstain and GPT models tending to hallucinate; and, counterintuitively, a shuffled incoherent haystack outperformed a logically coherent one on all 18 models. Thinking modes narrowed but did not close the focused-versus-full gap. The report ships with an MIT-licensed repository at github.com/chroma-core/context-rot carrying all three experiment families, reproducible against your own API keys.
At a Glance
- Topic
- Models
- Level
- Intermediate
- Format
- Paper
- Cost
- Free
- Duration
- ~40 min read, plus a reproducible codebase
- Provider
- Chroma
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Why longer context degrades accuracy even when the needed information is present
- ✓How needle-question semantic similarity, not just presence, drives retrieval failure rates
- ✓How distractors compound, and how the Claude and GPT families fail differently
- ✓Why a logically coherent haystack can score worse than a shuffled one
- ✓Read LongMemEval results comparing full conversations against focused relevant extracts
- ✓Design a minimal repeated-words probe that isolates length from task difficulty
- ✓Reproduce all three experiment families from the MIT-licensed companion repository
Highlights
- •Tests 18 models across four vendors rather than one, so the effect is shown to be general rather than a single model's quirk
- •The shuffled-haystack result is genuinely counterintuitive and reverses a common assumption about ordering retrieved documents
- •Every experiment ships as runnable MIT-licensed code at github.com/chroma-core/context-rot (301 stars), datasets included
- •Reached 260 points and 59 comments on Hacker News with an author participating, so the discussion is worth reading alongside it
- •Reports abstention versus hallucination behavior per model family, which is directly actionable when picking a model for long-context work
Who It’s For
Best For
- ✓RAG engineers deciding between a longer context window and better retrieval
- ✓Anyone building agents whose context accumulates across a long session
- ✓Teams evaluating whether a million-token window actually solves their problem
- ✓Researchers looking for a reproducible long-context benchmark to extend
Prerequisites
- •Familiarity with Needle in a Haystack style long-context evaluation
- •Basic understanding of embeddings and semantic similarity
- •Python plus OpenAI, Anthropic or Google API keys to rerun the experiments
FAQ
What is Context Rot: How Increasing Input Tokens Impacts LLM Performance?
A July 2025 technical report from Chroma testing 18 frontier models on how performance changes purely as input length grows. It extends Needle in a Haystack with semantic matching and distractors, runs LongMemEval focused-versus-full prompts, and adds a trivially simple repeated-words task. The finding that matters for RAG: what is in the context matters less than how much surrounds it.
Is Context Rot: How Increasing Input Tokens Impacts LLM Performance free?
Context Rot: How Increasing Input Tokens Impacts LLM Performance is free to access.
What level is Context Rot: How Increasing Input Tokens Impacts LLM Performance for?
Context Rot: How Increasing Input Tokens Impacts LLM Performance is aimed at a intermediate audience. Recommended background: Familiarity with Needle in a Haystack style long-context evaluation, Basic understanding of embeddings and semantic similarity, Python plus OpenAI, Anthropic or Google API keys to rerun the experiments.
How long does Context Rot: How Increasing Input Tokens Impacts LLM Performance take?
Expect roughly ~40 min read, plus a reproducible codebase. Most learners work through it at their own pace.
What will I learn from Context Rot: How Increasing Input Tokens Impacts LLM Performance?
You'll learn: Why longer context degrades accuracy even when the needed information is present; How needle-question semantic similarity, not just presence, drives retrieval failure rates; How distractors compound, and how the Claude and GPT families fail differently; Why a logically coherent haystack can score worse than a shuffled one; Read LongMemEval results comparing full conversations against focused relevant extracts; Design a minimal repeated-words probe that isolates length from task difficulty; Reproduce all three experiment families from the MIT-licensed companion repository.
Topics
Sources
This page was written from 2 sources, 1 on domains other than trychroma.com.