Context Rot

[ML 2.c] Needle in the Haystack: Embedding Training and Context Rot

By Archit Sharma

13 Min Read

You’ve probably experienced this: you paste a 50-page document into ChatGPT or Claude, ask a specific question about something buried on page 37, and the model either ignores it, gives a vague answer, or confidently cites something from page 2…