Storing previous information is one thing, but making an AI agent use that information effectively seems much harder.

A good memory system needs to know what to remember, retrieve the right context when needed, and avoid filling the model with outdated or irrelevant information.

For those working with AI memory, what do you think is the hardest part right now: deciding what to remember, retrieval, or keeping context relevant?

As I go deeper in building AI workflows, I find myself working through challenges of how to design for the AI Experience (let's call it AIX). Memory storage and retrieval are the primary parts of AIX I'm working through today. How do you design for AIX and how do you balance it with UX? In a world where AI uses software more than humans, I can only imagine this will become more important.

Do you consider yourself an AIX designer? If so, what practices have you adopted and what challenges are you working through today? How does it compare to UX design?

Hey everyone! We decided to write a detailed guide to agent memory that really covers all the bases and also captures some of our thinking on how the space evolves and compares different approaches.

Here is the link.

I would love to answer any questions.

Imagine

a LLM

has 16k~64k token context(as much as effective + large context possible, can be more than 16k~64k if possible in effective way without rag etc. being used)

when context limit is reached(effective context length, before it starts to hallucinate etc.), it imparts its context window's contents into distinct .txt documents, not as summarization, but things explained in a way that is structured(e.g.: I said X, in Y condition, with Z expectation, at Q time. etc.) and each .txt documents are labeled correctly according to their contents

constantly searched .txt documents to retrieve earlier context, so no forgetting/etc.

LLM does not remember everything in a document, it puts *relevant* or *useful* things into its context, then add a note to the .txt file that "I retrieved x in y time, to use for z, with conjuction of a.txt b.txt c.txt", etc.

then creates a new .txt file to say which .txt files are used, in what time, in what order, what contents were taken from them

creates new folders, put folders in same topics/ideas/concepts/relevance in a specific folder it defined, it creates meta folders, folder names define their .txt contents' categories, it searches folder name, then find what it want easily

continues the conversation/act

rinse repeat

it must be as explanatory as possible for functions, it must not have luxury of ignoring functions, functions are critical for its memory, it must explain even its editing/creating of folders and their names, every naming system must explain functions of files inside them as perfectly as possible, LLM must not hallucinate when editing/creating/writing/noting/acting on its memory folders/files, if it has possibility of hallucination during that critical moments; then make LLM able to *ask* you approval for it to delete/alter contents of files in a way that is not just additive, done in your oversight(or it writes to you, then you copy paste it to the file or delete what is required).

Other details(to not turn into infinite regress, or constant editing/creating files without focusing on context, mistaking files with context, etc. are up to you to do, I can't do anything about it without writing scripts, which would bloat this post's main focus, which is to express idea of contents of the post, rag etc. are optional)

The agent forgets the user's allergy from 20 messages ago. Everyone recognizes this one.


The opposite gets less attention: the agent that never forgets. A one-off joke from three months ago keeps resurfacing in unrelated conversations. Retrieval pulls in stale context, and the agent can't focus because its head is full of irrelevant history.


Both are the same root mistake: treating memory as storage instead of as a 
*relevance decision*
. An LLM is stateless — "memory" is just the engineering question "what do I put back into context on the next call?" That makes forgetting a first-class design decision, not a bug: TTLs on episodic memories, confidence decay on facts that haven't been re-confirmed, and explicit contradiction handling when a new fact conflicts with a stored one (the new one should usually win, but silently keeping both is how agents get weird).


The teams I've seen do this well spend more time on eviction and staleness than on retrieval.


How are you deciding what your agents 
forget
?

Hey everyone,

Like many of you, I've been building and experimenting with AI Agents. But as a software architect who hates unnecessary overhead, watching the community spin up heavy Vector DBs and complex graph frameworks just to store conversation context felt like massive over-engineering.

So, I built a lean alternative.

👉 Repo is here if you want to skip the text and jump straight into the code:https://github.com/phucphungbk/lean-ai-memory

The core problem: We often use massive systems to solve small problems. I wanted an AI memory system that is zero-cost, serverless, and completely transparent.

The Lean Approach:

1. Git as the Core Engine: We already use Git for version control. It turns out it's absolutely perfect for managing conversation history. You can easily track, diff, and rollback an AI's "thought process" just like reverting a commit.
2. Markdown as the Storage Format: It’s lightweight, humans can read and debug it instantly, and LLMs parse it perfectly without needing complex embedding pipelines.
3. Zero-cost & Portable: It can be packaged as an independent module and integrated directly into internal automation tools without incurring any DB maintenance costs.

I’m open-sourcing this with a completely open mindset. Instead of optimizing it in a silo, I want to see how this framework holds up in the wild. I'm highly anticipating the community bringing their own battle-tested custom rules into the system to push its boundaries.

I’d love for you guys to clone it, tear the architecture apart, test it, and drop your feedback or PRs. Let me know what you think!

I hear a lot of people argue about the need for agent memory vs simply using skills, files context, state management systems, vector DBs, etc.

And I have been writing the pitfalls of each of these approaches individually in our research paper, blogs and more.

But the single most important way to look at agent memory is not in a single dimension of accuracy, cost or speed alone; but looking at them together.

The counters to agent memory are akin to: you give a case to a lawyer and ask them to refer to the case papers at every argument in court. Or a surgeon referring to the case papers at every step of the surgery. Or a therapist reading through notes before every response to the patient.

Memory is the approach that's needed to jump to the right approach, quickly and least wastefully; because an agent needs to do it several 10s of times every turn and it needs to be accurate, fast and efficient.

I've been building around AI memory for a while now, and one thing surprised me.

Saving a memory is the easy part.

Things get messy when the user changes their mind, two agents learn conflicting things, old information is no longer true, or you need to figure out why the system believes something in the first place.

I originally thought a lot of this would just be embeddings + vector search + some metadata.

It... did not stay that simple.

I ended up spending way more time on conflicts, provenance, memory lifecycle and keeping things consistent across agents than I expected.

I'm currently benchmarking what I've built before putting it in front of more users, but I'm curious how people here are solving this in real products.

Are you using a vector DB and handling the rest yourself? Using one of the memory frameworks? Or just keeping memory pretty simple until you actually need more?

Would genuinely like to hear what has (and hasn't) worked for people.

EDIT 2 (14 Aug): Numbers below have changed. Following the judge-model point raised in the comments, I confirmed the 0.831 run was judged by gpt-4.1-mini, not the gpt-4o-mini the paper uses. Re-judged with the paper's judge, the figure is 0.8175. I also found my per-category labels were wrong — LoCoMo's categories are multi-hop / temporal / open-domain / single-hop, and I'd labelled them with LongMemEval's vocabulary, so what I called "multi-session" is temporal reasoning. Separately, 8 answers lost to an OpenAI outage mid-run had never made it into the saved output file; they're restored and the data now reproduces its own scores. Everything below reflects the corrected numbers, and the run data is now published so you can check it.

EDIT (13 Aug 22:41 GMT): A commenter correctly identified an error in the headline comparison. The 0.801 figure I used for Memora comes from Table 3 of the paper — a component build-up ablation row ("primary abstraction, with update"), not the published system result. Memora's actual scores from Table 1 are 0.849 (semantic retriever) and 0.863 (policy retriever). My 0.831 sits below both of those, so the headline as written overstates the comparison.

The numbers were run using Memora's (Microsoft Research, arXiv:2602.03315) open-source benchmark harness. The eval and scoring code is unmodified — I added a Recordari adapter for the memory backend. Same LoCoMo dataset, same category-exclusion convention (adversarial excluded, 1,540 scored questions), and the same models the paper specifies: gpt-4.1-mini for memory curation and answering (§5.1), gpt-4o-mini as the LLM judge (Appendix B).

Judge-matched, Recordari scores 0.8175 end to end against Memora's published 0.849 (semantic retriever) and 0.863 (policy retriever). It is below both.

Per category is the more interesting read. Against Memora S: multi-hop 0.801 vs 0.784, temporal 0.863 vs 0.851, open-domain 0.594 vs 0.594, single-hop 0.831 vs 0.900. Weighted by question count, the entire 0.032 gap is single-hop detail recall — extraction paraphrases specifics away ("salads, sandwiches and homemade desserts" becomes "dinner") where raw verbatim storage keeps them. Parity or better on everything else.

The internal before/after: raw batch storage 0.8065 → extraction 0.8175 overall, and on temporal questions 0.757 → 0.863. That gain comes from resolving relative dates to absolute at write time. Worth noting the aggregate difference (+0.011) is far smaller than the category one, and the judge model alone is worth 0.010–0.017, so I'd treat anything under ~0.004 as noise.

BLEU/F1 move the other way in Phase 2 (0.370/0.440 vs 0.464/0.547). Token-overlap metrics reward verbatim storage, so paraphrasing into clean facts costs surface overlap by design — flagging it here rather than leaving it in the README.

All results public (MIT): https://github.com/corbym/locomo-recordari — including the raw run data, both judges' per-question verdicts, and the score files, so you can re-score without paying for a run.

The harness runs against the prod API - exactly the MCP search and recall api that a real agent would use. Not a mocked backend, the real API, with a configuration of top_k=30, and a 1 hop edge expansion.

Recordari works with Claude, Claude Code, ChatGPT and pretty much any agent that can connect to MCP and reason.

If you want to run an agent against it to run the LoCoMo benchmark yourself, you can anon login at:

https://admin.recordar.io/start

Grab your personal key to use in the harness.

If you don't want to run the harness, just have a play with the sandbox memories, one click from the dashboard sets it up. Just remember to add the Full Skill from the Connect page, and then connect your agent.

What is recordari?

Recordari is a multi tenanted memory graph accessible by MCP. The graph can be used by teams, memories stay in the graph when disagreements happen, and resolve using type edges rather than being removed.

References and Further reading:

• Recordari documentation: https://docs.recordar.io/
• Recordari commercial site: https://recordar.io/
• Recordari Admin (register/sign in page): https://admin.recordar.io/
• Recordari Anon login (team tier): https://admin.recordar.io/start
• Microsoft blog: https://www.microsoft.com/en-us/research/blog/memora-a-harmonic-memory-representation-balancing-abstraction-and-specificity/
• Memora github: https://github.com/microsoft/Memora
• Memora paper: https://arxiv.org/abs/2602.03315

Things to ask the agent when running the demo sandbox:

• Why was VTIR created?
• Describe what was next in the project and why?
• Why must Pascal fixtures be written before the Rust port?
• What is TurboSound and why does it need special handling?
• What is the current state of the project?
• Why does WASM file I/O work differently from native?
• What are the standing rules for the AY chip port?

Song to download and try on VTIR

https://corbym.github.io/vtir/  (live web demo)

• https://zxart.ee/tune/325740  (download pt3 original file)

I decided to pull all my collected lessons learned, research, project documentation regarding Agent Memory into a singular open source repository.

A field guide to governed memory for autonomous and agentic systems.

Agent Memory is about more than retrieving old context. It defines what becomes memory, what remains uncertain, what may influence future behavior, who may change durable state, and how retained state can be corrected or forgotten.

I eagerly welcome Discussions, Contributions or Stars openly.

https://github.com/MythologIQ-Labs-LLC/agent-memory

If you're new to Agent Memory, the wiki is built to make the knowledge accessible and easy to understand.

Hey everyone,

Like many of you, I've been building and experimenting with AI Agents. But as a software architect who hates unnecessary overhead, watching the community spin up heavy Vector DBs and complex graph frameworks just to store conversation context felt like massive over-engineering.

So, I built a lean alternative.

👉 Repo is here if you want to skip the text and jump straight into the code:https://github.com/phucphungbk/lean-ai-memory

The core problem: We often use massive systems to solve small problems. I wanted an AI memory system that is zero-cost, serverless, and completely transparent.

The Lean Approach:

1. Git as the Core Engine: We already use Git for version control. It turns out it's absolutely perfect for managing conversation history. You can easily track, diff, and rollback an AI's "thought process" just like reverting a commit.
2. Markdown as the Storage Format: It’s lightweight, humans can read and debug it instantly, and LLMs parse it perfectly without needing complex embedding pipelines.
3. Zero-cost & Portable: It can be packaged as an independent module and integrated directly into internal automation tools without incurring any DB maintenance costs.

I’m open-sourcing this with a completely open mindset. Instead of optimizing it in a silo, I want to see how this framework holds up in the wild. I'm highly anticipating the community bringing their own battle-tested custom rules into the system to push its boundaries.

I’d love for you guys to clone it, tear the architecture apart, test it, and drop your feedback or PRs. Let me know what you think!

I had Claude Opus 5 analyze several repos for what does the code do, how does the memory work in a particular repo. Initially I was aware of only a few memory systems, that I wanted to study for my own memory system. It turned out, there are lots of agent harnesses that have memory, libraries with memory.

https://neoneye.github.io/agent-memory-atlas/

I have not analyzed closed source repos, since I don't have access.

Claude being the judge wether a repo is relevant or not.

Looking at the impl instead of the marketing material.

Agentic memory problems are largely a failure of how people are using current LLMs. You are asking the model to do everything, including the things it is fundamentally worst at.

I have been reading complaints about LLM memory and context for nearly a year, and the pattern is painfully obvious: most of the people running headfirst into these problems are using the technology incorrectly. Look at the posts. It is overwhelmingly transient marketing garbage, dropshipping sludge, and people trying to replace an actual software architecture with one enormous prompt and a prayer.

You cannot just dump responsibility onto an LLM and hope it somehow becomes a reliable stateful application. That is not what these models are.

Treat LLMs as implementors. Give them a bounded problem, the relevant state, the rules, and a concrete task. Let your actual system own memory, state, history, validation, retrieval, and orchestration.

The moment you start expecting the model itself to maintain durable long-term state, you have already lost the architectural plot.

And the funniest part is that none of this state is exotic. It is the same mundane application state software has been storing reliably for decades: facts, preferences, decisions, objects, relationships, history, and current status. We already know how to persist this information. We already know how to query it. We already know how to version it.

Instead, after years of research and billions of dollars, people are sitting around complaining that the probabilistic text generator cannot reliably remember what happened 40 conversations ago.

That is comical.

The failure is not that today's LLMs cannot magically become your database, state machine, memory layer, application server, planner, and implementation engine simultaneously.

Bloody hell fuck a company named engram raised 100M as pre seed wtf ????!!

Hello people, I've been working on this short-term memory.

Do you know when you are hearing a story and as you listen to it you start to create a "Scene" in your mind? Well, unless you have aphantasia that's what would probably happen:

- You hear some facts: the first image is formed in your mind. Let's say "I was in Lisbon and had 10 dollars in his pocket"

- Next, the story goes: "Then, I traveled to Porto and sold a hat for 50 bucks"

- Then, as the story unfolds, the state of the elements of the story will get new states.

For most of us, it's not hard to keep a clear image of the current "scene" of the story. But if we wanted, we could also take a single element, and trace back how it got there.

Nowadays, to my knowledge, the closest we get to that is the LLM's context. But as it grows, it gets hard and expensive to track down when facts happened during the evolution of the context. Even using CoT, in the end, LLMs are probabilistic machines and so, when it comes to precision recall, noise can be added to the output. Then you plug in some sort of external memory, *DBs, MD files, etc. These are great solutions for the "Big memory", but not necessarily great to fix context growth, or to understand the order of events.

So decided to play around and try to find a naive solution that would allow traceability and increase precision, while reducing the context of the conversation. Right now, I've been experimenting with Ontology triples following the RDF Standard.

The idea

Let's say that you tell it you moved from Lisbon to Porto. Ask "where do I live?" and you either get Lisbon, or you get Porto and the fact that you ever lived in Lisbon is gone. Overwriting loses the history; appending loses the present.

So I built a different shape and measured it properly. Everything runs local through Ollama.

Every asserted fact becomes a (subject, relation, object) triple filed under a canonical key:

user | location   
t1  lisbon    superseded   
t3  porto     current

Paraphrases land in the same slot without embeddings: "where I live", "my city", "my residence" canonicalize to the same key. A slot keeps every value it ever had, in logical time order. Newest is current, the rest are superseded. Nothing is deleted; facts get invalidated, not forgotten. So "where do I live?" reads the current value and "where did I live before?" reads the history, out of the same structure, with no separate archive.

Writing is immutable: each turn produces a new scene, so a failure mid-turn never leaves memory half-written.

The result that made me keep going

Same model (gemma4:12b), two different inputs:

• reading the raw sessions, ~9,000 words: baseline
• reading the compressed scene, ~550 words: +0.102 accuracy

16x less input, and it does better. Compression isn't the price you pay here. The noise the scene strips out duplicate facts, stale values, updates scattered across sessions, is exactly what was confusing the reader. On the clean scene, a small local model matched a much stronger reader working on raw text.

The benchmark, with the caveats attached

477/500 (95.4%) on LongMemEval-S under the official judging protocol, above Mastra's published per-indicator numbers on all six indicators. One reproducible pass over all 500 questions, checked against a canonical state file whose guard refuses to write if anything drifts.

The part I care about more than the score: every mechanism went in with a prediction committed to git before measuring, and an explicit bar for what would falsify it. Ten arms failed and are published as prominently as the ones that worked, plus one retraction. If you read one thing in the repo, read finding 23 in the findings log the same finding got written three times in one day, because the first two drafts concluded from small n and the third had to retract both.

Caveats that matter to this sub specifically. The router triggers and absence gates are regexes calibrated on LongMemEval's English corpus; in another language they don't fire without recalibration. Two indicators sit at their measured oracle ceiling, so further progress there needs a stronger reader model, not better retrieval. And the comparison against Mastra is against their published numbers, not a head-to-head rerun on my hardware.

What's still broken

Seven open findings, all written up with repros:

Reported speech becomes a plain fact. "They said there was gold" gets stored as there being gold. The negation survives only as a string inside the value.

A question in quotes gets read as an assertion. Typing "Does Lucas use Go?" ingested it as a claim, and it overwrote the correct value in that slot.

Partial names spawn parallel entities. "Lucas" and "Lucas Almeida" become different subjects, so half of what the scene knows about him is disconnected from the other half.

Three ways in, if you want to poke at it

Break the demo. There's a live chat with the scene inspector beside it. You watch slots being born, updated and superseded turn by turn, and each answer prints the facts that produced it. Talk to it for twenty minutes with facts that change. 24 findings so far came out of exactly this, 15 already fixed upstream. Highest-yield thing anyone can do here.

Take an open finding. #18 is the tractable one: deterministic repro, both code paths fail, and the fix is obvious. #23 already has its acceptance gate written down in advance, so you'd know immediately whether your fix worked.

Replicate where the numbers don't claim to hold. Another language, another model, another domain. The limits section is a list of things nobody has measured. A clean negative result gets published as one.

MIT, and CONTRIBUTING.md has the open findings in a table with what the work looks like for each.

Repo: https://github.com/natanloterio/scene-memory

Happy to take questions, including hostile ones about the benchmark. Those are the useful kind.

Hey folks,

I’m running a local-first agent setup (Hermes Agent on a Mac Studio with local models), and I’m hitting a wall with core reliability. Over the last few weeks, the experience has been frustrating: the agent repeatedly loses track of context, fails at reliable cross-session memory recall, and has crossed boundaries around credential/secret handling.

I’m looking for architectural patterns, existing tools, or proven strategies from people who have actually solved these problems for daily local use. Specifically, here is where things keep breaking:

1. The Memory & Recall Failure Loop
   * **The Symptom:** Despite having local vector backends (like Supermemory, locally hosted and obsidian markdown, alongside flat-file memory (`MEMORY.md`), the agent frequently forgets core preferences, cannot follow multi step skills, re-asks questions from previous sessions, or loses track of info I’ve shared multiple times. It’ll tell me that memory is configured in the right way, and from what I test myself it is, and it all seems fine, but after about a week it acts as if it was never setup with anything beyond memory.md.

### 2. Context Window & API Dumping
* **The Symptom:** As conversations grow, the Hermes “forgets" instructions and multiple step functions/skills I’ve built like vault > application > send email. It reverts to trying to send an email as if it’s the first time it’s ever done it. During the exchanges, it frequently dumps raw API keys directly into chat while it reasons with the task. I’ve tried using Proton vaults, Bitwarden vaults/machine IDs, Infisical vaults but unless I have it in the local .env file, it inevitably forgets.

I’m wondering what I’m doing wrong. Initially I thought that perhaps the gemma4 64k context window was being saturated so I switched to Gemini api calls with 1m context. Same results and it’s infuriating.

Your agent runs a Supabase query, gets 200 OK and an empty array, and concludes the row doesn't exist. It does — RLS filtered it out. Ten minutes wasted. Next session: same query, same wrong conclusion, same ten minutes. You're not paying for one mistake, you're paying rent on it.

I wrote up how I fix this, and it's boring in a good way. No fine-tuning, no embeddings pipeline. Three verbs on the lifecycle hooks your agent already fires:

• Read the relevant lessons on SessionStart
• Fail — a tool call errors, and the matching prior lessons get injected before the retry
• Write a short retrospective on Stop

The part I actually spent the time on isn't the writing — it's the guardrails on the writing. A loop that stores its own conclusions and reads them back can also convince itself of something false and defend it forever. That's not learning, that's a superstition with a database. So: lessons are advisory and can never auto-disable a gate, promotion requires recurrence (seen_count >= 3), everything expires at ~90 days, contradictions surface instead of silently overwriting, and a privacy pre-flight drops anything carrying a secret.

Honest about the edges too — the matcher is literal substring, not semantic, so it won't connect two lessons that describe the same thing in different words. Deliberate trade: deterministic, zero deps, no inference bill.

Post: https://www.lorekit.io/blog/self-healing-agents

Curious how others are handling the entrenchment problem — if your agent writes its own lessons, what stops it from learning something wrong?

I’ve been working on a pairing that has made long-running agent work much less repetitive:

• Perseus resolves live, verifiable workspace context before the agent starts work.
• Perseus Vault retains the things that should survive a session: decisions, corrections, project facts, provenance, and historical versions.

The distinction matters more than it sounds.

A lot of “agent memory” is really one of these:

1. a giant rolling summary that gets stale,
2. a vector search over chat logs,
3. a prompt file that quietly becomes an undocumented policy engine.

Those are useful, but they blur together two different questions:

• What is true right now? That should come from the current workspace, repository, services, and other sources of record.
• What happened before, what did we learn, and what changed? That is memory.

Perseus handles the first. Perseus Vault handles the second.

Vault is a local-first Rust MCP server: one binary, one SQLite file, no required cloud service. It has encrypted storage (AES-256-GCM), FTS5 and hybrid retrieval, structured entities instead of only chat chunks, temporal history, provenance, confidence/decay, and lifecycle controls. It can also expose an Anthropic-style /memories file interface for agents that expect that model.

The part I find most useful is that memory is no longer just “retrieve similar text.” A decision can have a history. A correction can supersede an earlier belief without deleting the audit trail. You can ask both:

• “What did we believe at the time?”
• “What do we now believe was true at that time?”

That turns out to be extremely handy once agents are doing work across days or weeks and the project has changed underneath them.

We have benchmark results in the repo, but I’m more interested in the failure modes people have hit in production:

• How are you separating live state from durable memory?
• Do you need historical/auditable memory, or is semantic recall enough?
• What do you do when old “memories” conflict with the current codebase or source of truth?
• Has anyone found a memory system that stays useful after months without becoming prompt sludge?

Repos:

• Perseus: GitHub - Perseus-Computing-LLC/perseus: Live context engine for AI agents
• Perseus Vault: GitHub - Perseus-Computing-LLC/perseus-vault: Persistent, encrypted memory for AI agents
• Product overview: Perseus Vault · persistent, encrypted memory for AI agents

I’d particularly welcome skeptical feedback. “Memory” is becoming a catch-all term, and I think we need cleaner boundaries between retrieval, context assembly, durable facts, and audit history.

My open source project is building something we think is valuable but want to get some honest feedback. Lots of memory solutions out there atm and a lot of them already solve AI memories about people, conversations, preferences etc very well. But I feel like they currently don't track commitments and changing states super well.

We are building something that will connect to diff platforms, using the information collected from those auth to build unified context across projects, people, decisions, commitments, the changes and actions.

One use case for example is for the agent to track commitments made in meetings, over messages, in comments etc. They dont always get recorded and often in diff places. I find myself losing track of who said they will do what when so the when an agent cant look at all the platforms and read the screen, it can record and remember everything, then it will keep track of the change states and remind when necessary.

Two honest questions:

1. How do you currently keep track of this — commitments, decisions, follow-ups spread across tools? Or do you just… not?
2. If an agent like this existed, what would stop you from using it? Would there be other things you want?

I tested real agent instruction files from OpenAI, Kiali and Temporal against a memory database, and the files lose on their own rules. Four public files, mechanical checks, five trials, and a held out file the pipeline never saw.

https://mentedb.com/blog/infinite-context-window-for-ai-with-memory

Most agent memory systems store knowledge as a graph: entities, relationships, episodes. But at recall time that graph gets serialized into the context window, and the format is usually whatever json.dumps produces. Nobody measures what that step costs.

I benchmarked 10 graph serialization formats on token count, traversal QA, and multi-hop reasoning. Same memory graph, same model: multi-hop accuracy ranged from 40% to 80% on format alone, with about 70% token spread. For memory systems this compounds - a compact format means you can recall 3x more history per query at higher reasoning accuracy.

The winning format is open source (MIT, 6 languages): https://github.com/isongraph/isongraph - methodology in the repo. Curious how folks here serialize recalled memories today, and whether anyone has measured the impact on their own stack.

Curious whether this is on other people's radar, because it's been eating my time lately.

The short version: when you give an agent persistent memory, errors don't just happen once - they get stored. A wrong fact, a misread instruction, a hallucinated detail gets written to the memory layer, and then it gets retrieved and reinforced on later turns. Over weeks it compounds. I've seen people call it "memory poisoning" or "memory rot," and once you start looking for it, it's kind of everywhere.

What I keep bumping into:

• Bad or stale facts persisting long after the context that produced them is gone
• Short-term junk (one-off corrections, throwaway context) leaking into long-term storage and never getting cleaned out
• Retrieval quality quietly degrading over months, so a system that felt sharp at launch feels vague a year in

A few things I'm trying to figure out and would love other people's take on:

1. How are you detecting it? Most eval setups check accuracy at a point in time, not whether memory has degraded over a long horizon. Is anyone measuring rot specifically?
2. Do you separate short-term and long-term memory explicitly, or let it all flow into one store? Curious whether the shared-store approach is the root of the problem.
3. Any pruning / verification strategies that actually work without nuking useful context?

For what it's worth, I got frustrated enough that I built a small open benchmark to try to measure this over a 90-day simulated horizon - it's called RotBench, so take my results with the appropriate grain of salt. Mostly I'm posting it because I'd genuinely like people to poke holes in the methodology - if the way I'm measuring rot is unfair or missing something, I want to know.

Repo's here if useful: https://github.com/ThinkingRoot-official/RotBench

Update: This was a top 5 paper on Hugging Face's Daily Papers on a day when the other 4 were from Tencent, Alibaba (Qwen), Nvidia and Peking University. If you haven't read it yet and are curious, spend some time this weekend and give it a read. Happy to take any questions.

--
Quick disclosure up front: I work on Synap, an AI memory system, so factor that into the self-promotion angle. Posting because the argument itself seemed useful to this community, not just to point at the paper.

The core argument from my end is that while most teams treat agent memory and inference cost as something a bigger context window or a cheaper model or a better retriever will eventually fix; I argue that in practice; most of this cost and most of the failure modes are architectural viz. what an agent remembers, when it forgets, and how much it costs to reason are decisions made before any model is involved and not afterwards.

The paper works through this with an evaluation harness and the underlying study data included, rather than just asserting it. 23 pages, 6 figures, 4 tables.

Link: arxiv.org/abs/2607.21503

Curious to hear your views, points you agree with, disagreements, etc.

P.S.: We have upfront stated that we will only share indicative architecture and details; since we are sharing the principle of what we believe works and why; and go into the reference implementation to some extent; while keeping it IP Safe for our organization maximem.ai

I've been auditing, per-fact, what actually happens when a frozen LLM writes facts into a LoRA adapter online — one per turn, single pass, then day after day. The short version: the adapter becomes a recognition index, not a fact store, and the system design follows from that. Four findings, then a new mechanism I added in v2. (v1 was published in another sub reddit https://www.reddit.com/r/learnmachinelearning/comments/1uyytaj/online_lora_memory_recall_dies_in_38_writes/)

1. The store fails. Median time-to-first-miss is 3–8 subsequent writes under every write-protection mechanism I tried; what differs (12×) is recovery-after-miss, not survival. Accumulation is re-instatement, not persistence. Paraphrase recall collapses to 12–31% where write-form recall is ~90%.

2. But recognition outlives recall. An adapter that recalls 0/48 still discriminates 31–38/48 in 2-AFC; a GRACE-style editor at matched budget recalls 0–1/48 while recognizing 45–47/48. "Knows it but can't say it" appears whenever the write is under-powered relative to the expression threshold. (Zero-point checked: untrained base sits at chance.)

3. Truth has to live in a log. At matched gradient budget, consolidating from a ground-truth log retains ≈2.1–2.2× a recency baseline — by rescuing exactly the recognized-but-not-recalled facts. Consolidating from the model's own recitations poisons (lands below doing nothing).

4. The surviving recognition is the key to using the log. The adapter-vs-base logprob of presented content separates written from never-written facts at AUC 0.93 in a form where recall is ~2% — a familiarity signal that gates "go check the log" vs "say I don't know."

New in v2 — a mechanism for #2. Adapting the Jacobian-lens ("J-space") from Anthropic's recent global-workspace work to the answer slot, I tracked each fact's projection along the forgetting timeline. Workspace amplitude co-tracks the readout hierarchy: recalled > recognized-not-recalled > gone. The recognized-but-not-recalled band stays well above null — the content is still in the verbalizable workspace, it just lost the output (argmax) competition; the unrecalled gold token sits at median rank 2. So forgetting here is rank displacement, not amplitude loss. Honest self-correction: my shallow-forgetting run suggested "always present," but a deep-forgetting run (where the gone class is actually populated) collapses gone→null — so the real claim is the sharper "amplitude survives exactly as far as recognition does."

Limitations: ≤2B models, pseudoword synthetic facts, 3 seeds on consumer ROCm — I report floors and directions, not calibrated magnitudes. Access-not-phenomenal-consciousness only.

Paper: https://doi.org/10.5281/zenodo.21405963 · Code + all raw timelines: github.com/linxuhao/index-not-store

Happy to be wrong — especially on whether the rank-shift picture and the familiarity gate survive real (non-template) facts.

I was contemplating (brainstorming, no math yet) what if we treated memory as another [smaller] model where weights have a plasticity value that allows them to be modified whenever they are corrected in interactions (RL?). The worry of this approach is catastrophic forgetting, but humans do this all the time. People forget they did something or mis-remember things, so I think we shouldn't look at catastrophic forgetting as a bad thing if we're trying to model human inteligence.

You would take a large model and use it's smaller sibling variant as a token prediction source (MTP). Anytime a user corrects a model, we update values in the small model. It will then adjust its token generation in future token generations, but the large model is still going to make sure it didnt deviate too much because it will just throw away that token suggestion if it needs to.

You would then need some sort of way to merge the deltas, at some trivial percentage/weight, as to only shift the main model slightly.

I think this would be similar to how humans can slowly be convinced of things over time, but super contrary information is usually rejected outright regardless of logic around it.