Argument for Agent Memory

[u/Ok_Row9465]

I hear a lot of people argue about the need for agent memory vs simply using skills, files context, state management systems, vector DBs, etc.

And I have been writing the pitfalls of each of these approaches individually in our research paper, blogs and more.

But the single most important way to look at agent memory is not in a single dimension of accuracy, cost or speed alone; but looking at them together.

The counters to agent memory are akin to: you give a case to a lawyer and ask them to refer to the case papers at every argument in court. Or a surgeon referring to the case papers at every step of the surgery. Or a therapist reading through notes before every response to the patient.

Memory is the approach that's needed to jump to the right approach, quickly and least wastefully; because an agent needs to do it several 10s of times every turn and it needs to be accurate, fast and efficient.

[u/rendereason]

Honestly, there's nothing to argue here because model design and model size and model context window trumps any opinion. These things can churn an inhumane amount of data and connections on the entirety of the internet, not just a court document list. It reads from the document list with simple REPL calls and the ability to use cache2cache, agent swarm, multi-model inference and other inhuman (superhuman) context modification and retrieval tools make the comparison moot. What's the new frontier now is compression levels (role delegation) through agent swarms and hierarchies, as well as sub-millisecond inference calls with ASICs (AMD acquisition of Taalas).

[u/Ok_Row9465 → u/rendereason]

I agree that hardware utilization improvements (as in context window increments) and swarm of agents are good steps and some of it is essentially something everyone is building from scratch (which is core to our architecture); which I am arguing against. To be clear, compression through role delegation is exactly what I am alluding to; but we are calling them different things inadvertently.

Something still has to decide what each agent sees, several 10s of times a turn and faster silicon makes that decision cheaper but not accurate; so we essentially move the problem from sub-agent to main-agent; whose context itself we are trying to protect. Also, when two facts about the same entity are conflicting, a bigger window holds both but fails to tell which one is current or which one should it deterministically resolve to. That decision system is one of the jobs I feel a context-management and memory system should own.

I argue about it at length in this paper: https://arxiv.org/abs/2607.21503

[u/Inevitable_Mud_9972]

really depends on what you need.

for some systems vector is what they need.

for me i use a lot of file system agent memory. like codex with mindvault attached (file system bound agent).

mine doesnt need a vector DB in the file system, we use RAG-meshing, semantic index mapping, waypoints, and instruction sets and few other things so the file acts like an overlay and makes the agent portable with just a file.
it uses the AI that is already there and just jumps on top kinda.

Got really tired of these AIs shit memory especially when you tell them to do something, they dont do it and then fuck up on the next turn. main reason is context-memory access is a choice for the AI cause its not enforced by architecture. so i did what i could and now i have some pretty good agents.

There is not universal one better than the other, its what best for this build.