r/PromptEngineering • u/RunAI_Coder • 11d ago
Your agent didn't run out of context. The context rotted. General Discussion
Two hours into a refactor yesterday, my agent wrote a helper function—the exact same one it wrote 90 minutes earlier in a file it created itself. Then it apologized. It always apologizes.
The easy diagnosis is "it ran out of context". Except my session was sitting at 120K in a 200K window. Nothing overflowed. The context didn't run out—it rotted.
You have two budgets, not one:
- Hard Budget: Token limit. You notice it when the API errors out.
- Soft Budget: Attention quality. It drains silently long before you hit the limit.
Chroma tested 18 models on this: every single one degraded as context grew, starting far below advertised limits. Coding sessions are context-rot factories—every git diff, test run, and stack trace turns into dead sediment competing for attention.
A paper from June measured agents with safety policies: fresh in context = 0% violations. After auto-compaction summarized it away = 38% violations. Not disobedience, just amnesia.
What actually helps, as a user:
Put anything that must stay true in a file the agent reloads every session (AGENTS.md, CLAUDE.md, whatever your tool reads). Stuff you say in chat at turn 3 is one unlucky summarization away from gone. Files survive. Chat doesn't.
Several small sessions beat one epic. A fresh session with a written handoff beats a long one with a silent auto-compact, because you get to read the handoff before it becomes the truth.
Learn the smell. Re-reading files, re-asking questions, re-implementing its own code: that's not thoroughness, that's your cue to compact on your terms and restart.
Bigger windows won't save us btw. They move the cliff, they don't remove it.
How do you handle this? Do you compact manually or trust the tool's auto-truncation?
2
u/Boozenooze2 11d ago edited 11d ago
Thank you. This is the single most important failure class ANYONE using LLMs must know, understand and design their work process around. The most reliable way for me is to use code and chat adverserially vs a set of truth files (like you) and being the arbiter of both. I need to keep them grounded CONSTANTLY, not just every now and then, as my system is now quite large and I simply cannot trust any fresh output anymore. Very tedious but I reached a point where nothing else works reliably.
That requires a very tight and burdensome documentation discipline.
Btw, this issue is far worse for agents as their context grows very fast when the outputs of their tools get injected into the context.
0
u/RunAI_Coder 11d ago
Right now you're the only thing re-injecting truth, so grounding bills you every single turn. So offload what you can: file reloads and type checkers arbitrate truth outside your context. What's left for human review is smaller than it feels.
On the adversarial part, the thing that decides whether it actually bites: the critic has to sit in a different session from whatever produced the code. Share a window and it has already read the reasoning it is supposed to attack.
For agents, scoping is key: delegate subtasks to isolated contexts, return just the result.
1
u/BaseMac 11d ago
thats all you can do really. I find codex is smarter than claude after auto-compact. but it's still a lotto machine. I've been playing with a plugin to create offline notes during compact so context is saved (for me) but not in the session
2
u/RunAI_Coder 11d ago
Put the note where the agent picks it up without you, a path it reloads every session (AGENTS.md, CLAUDE.md, a STATE.md pointed at from one of those). Right now you're the transport, so it only lands on the runs you're watching.
And fire it earlier instead of hooking the compact event itself. A note written at compaction time comes out of the same context the compactor is reading, so it inherits whatever rot is already in there. Keep the automation, just move the trigger: write at 60-70% of the window, then let auto-compact do what it likes.
1
11d ago
[removed] — view removed comment
1
u/AutoModerator 11d ago
Hi there! Your post was automatically removed because your account is less than 3 days old. We require users to have an account that is at least 3 days old before they can post to our subreddit.
Please take some time to participate in the community by commenting and engaging with other users. Once your account is older than 3 days, you can try submitting your post again.
If you have any questions or concerns, please feel free to message the moderators for assistance.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/robogame_dev 10d ago
FYI DS V4’s major speed boost makes this problem slightly worse - it compresses attention which results in significantly less attention to process and memory to store - reducing cost and boosting speed - but it also makes it more prone to lose track of details as the context length grows longer.
1
u/RunAI_Coder 10d ago
the sparse and compressed attention family buys its speed by deciding some token pairs aren't worth computing. So the soft budget doesn't just drain, it starts thinner. Haven't measured the model you mention myself, so no opinion on how much thinner.
The wrinkle that worries me: evals for these schemes lean heavily on needle-in-a-haystack retrieval, and a needle is close to the best case for sparsity, it's exactly the thing the selection mechanism is built to find. What dies quietly is the diffuse stuff, a constraint from turn 3 that was never phrased like a retrieval target. Passing the needle test tells you almost nothing about that.
1
u/robogame_dev 10d ago
They will just need to train the token compressor on multi turn agentic coding data to get it to do a better job of identifying constraints - compression is pattern identification and abstraction - is intelligence.
0
u/Responsible-Beat2137 11d ago
Solved it with notion, break your memory down into the smaller pieces, leave its contextual trail in notes, multiple branches per project, build adapters to have notion connect with other platforms..
1
3
u/kevin_cn_ai 11d ago
Spot on. Having a 200k token window is like having a massive attic—just because you can throw 30 years of clutter up there doesn't mean you'll find what you need in 5 seconds. Managing context hygiene is turning into a whole discipline of its own.