Context Rot: How Increasing Input Tokens Impacts LLM Performance
by Chroma
Empirical proof that long context isn't free — model reliability degrades as inputs grow.
Overview
Context Rot: How Increasing Input Tokens Impacts LLM Performance is a technical report published by Chroma on July 14, 2025, authored by Kelly Hong, Anton Troynikov, and Jeff Huber. Across controlled experiments on 18 state-of-the-art models — including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 — the authors show that models do not process context uniformly and that reliability drops as input length grows, even for tasks as simple as retrieval and text replication. Key findings: performance degrades faster when the needle and question are semantically (rather than lexically) similar; even a single distractor lowers accuracy, with uneven effects across models; conversational benchmarks (LongMemEval) show a large gap between focused prompts and full-history prompts; and models fail at literal text replication as sequences lengthen. The practical implication for AI engineers is context engineering — curating and compressing what goes into the window rather than maximizing it. An open-source toolkit for replicating the results is provided on GitHub.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- ~30-40 min read, long-form technical report
- Provider
- Chroma
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓Why longer input context does not equal better performance, with measured degradation curves
- ✓How needle-question semantic similarity and distractors accelerate accuracy loss
- ✓How full-history vs. focused prompts change results on LongMemEval-style conversational tasks
- ✓Why 'context engineering' (curating the window) beats naively maximizing context length
- ✓How to reproduce the benchmarks with Chroma's open-source toolkit
Highlights
- •Controlled evaluation across 18 frontier models (GPT-4.1, Claude 4, Gemini 2.5, Qwen3)
- •Goes beyond the overused Needle-in-a-Haystack test with harder, more realistic setups
- •Directly motivates RAG and retrieval over dumping everything into a long context
- •Fully open, with a replication toolkit on GitHub
Who It’s For
Best For
- ✓Engineers building RAG or long-context LLM applications
- ✓Agent developers managing context windows and memory
- ✓Anyone deciding between long-context prompting and retrieval
Prerequisites
- •Familiarity with LLM prompting and context windows
- •Basic understanding of retrieval / RAG concepts
FAQ
What is Context Rot: How Increasing Input Tokens Impacts LLM Performance?
A Chroma technical report (July 2025) showing empirically that LLMs do not use their context window uniformly: performance grows less reliable as input length increases, even on simple tasks. Essential reading for engineers building RAG, long-context, and agentic systems who assume that stuffing more into the prompt is safe.
Is Context Rot: How Increasing Input Tokens Impacts LLM Performance free?
Context Rot: How Increasing Input Tokens Impacts LLM Performance is free to access.
What level is Context Rot: How Increasing Input Tokens Impacts LLM Performance for?
Context Rot: How Increasing Input Tokens Impacts LLM Performance is aimed at a advanced audience. Recommended background: Familiarity with LLM prompting and context windows, Basic understanding of retrieval / RAG concepts.
How long does Context Rot: How Increasing Input Tokens Impacts LLM Performance take?
Expect roughly ~30-40 min read, long-form technical report. Most learners work through it at their own pace.
What will I learn from Context Rot: How Increasing Input Tokens Impacts LLM Performance?
You'll learn: Why longer input context does not equal better performance, with measured degradation curves; How needle-question semantic similarity and distractors accelerate accuracy loss; How full-history vs. focused prompts change results on LongMemEval-style conversational tasks; Why 'context engineering' (curating the window) beats naively maximizing context length; How to reproduce the benchmarks with Chroma's open-source toolkit.