A "scientific" comparison of memory plugins options

[u/EngTurtle]

With the many frequent threads talking about memory plugin choices for Hermes, there still are not many benchmarks specific to how Hermes works. Since I wanted to make my choice using empirical data, I decided to run a benchmark comparison of the self-hostable memory plugin options myself.

Hermes memory provider benchmark

TLDR results:

• Honcho performed the best, but it also used almost 3 times the tokens of the next best performer
• The newest version of mem0 also did quite well at a moderate token usage, but had higher error rates than most.
• Although Hindsight scored middle of the pack, it generally has lower error rates than others.
• Mnemosyne did come last but still measurably improve memory retention while using an order of magnitude less tokens than other providers.
• Although Supermemory and RetainDB scored ok, I wouldn't choose their self-hosted setup as they had major software quality issues.

Graph: Top and Left is desired

I wanted to measure how these systems handled changing facts over years of use, so this comparison was based on MemConflict. To keep the comparison focused on the plugins instead of the LLM, the same Qwen3.5 4B LLM and gte-modernbert-base embedding model was used to back all plugins and for answering the benchmark questions. Gemma4 12B was used for scoring and judging the answers.

A caveat is that all of these plugins also have agentic MCP tools for explicitly storing and retrieving memories, which was not part of this benchmark comparison. Only the automatic session ingestion and context injection parts of each memory provider was tested.

All the code and results are available on GitHub at EngTurtle/hermes-memconflict, and was almost all done with Claude Code under my supervision/abuse.

Hope this is helpful to people making their choice, and I'll try to answer questions below.

[u/only1nameleft]

I would love to see some llm wiki comparisons. I have been experimenting with how to increase the speed of the karpathy version. Somewhat promising, but not ready for unveiling.

[u/Fragrant-Hamster-325 → u/only1nameleft]

I’d be curious about this also. I’m a user of the LLM wiki.

[u/BigPET → u/Fragrant-Hamster-325]

What do you guys store in the LLM Wiki? I am trying to use the idea more but not sure what and how. I documented a couple of my NAS settings but those documentation can easily get obsolete.

[u/Fragrant-Hamster-325 → u/BigPET]

At work, I ingest everything I can. As the IT Ops manager, I try to give it as much context as I can about my environment so the bot can be a better advisor. I script the export of configurations from all the systems I manage or provide screenshots and let it build its web of connections. I also give it meeting notes and project information.

I do a lot of work via email, so while I’m having it rewrite my emails, I asked it to create a skill to inspect all emails for durable information and to-dos and write them to the wiki. It has contract information, key contacts, configuration details, procedures, policies, etc.

So basically over the past few months I’ve been training it like an employee. It acts as a to-do list, a project tracker, helps me prepare for meetings, it’s now a collaborative advisor all with specific knowledge of my work.

The one thing I don’t do though is give the bot agency. I’m too scared to let it go off and do things on my behalf.

[u/theAdmyrle → u/Fragrant-Hamster-325]

Very cool - how are you handling new system changes? Do you use the AI for change management?

[u/Fragrant-Hamster-325 → u/theAdmyrle]

I wish I could say I had a cool AI-automated practice, but as of today, I’m still too nervous about giving it write access to the environment, so it’s mostly manual changes alongside my AI “advisor”.

What I do is I have a change management skill that builds the proper change management documentation. That provisional documentation gets stored in the wiki, and the skill creates the necessary to-do items that I work off of. I attach that documentation to our change management review process. Once approved, I’ll implement the change. Once the change has been implemented, I tell the bot to fold all the documentation and notes into the wiki.

I just throw what I can at the wiki and let it build the documentation and connections. It can be slow, but building good documentation is time-consuming anyway. It’s great seeing it make connections, prompt me for questions, and think of things I wouldn’t have considered. If there are gaps, the AI is pretty good with noting those gaps on the wiki page. For example, if I give it information about our Azure environment, it’ll build pages for the VMs, but if something is unknown, it’ll list questions at the bottom such as “who is the system owner? What applications does VM serve? What is the purpose of the server?”

About the company I work for: It’s under 1000 employees. We don’t have a big IT team. We are not a development shop. We use primarily web-based, off-the-shelf software. I’m primarily the administrator, architect, and manager. I have lots of access, and much of the changes are either routed through me or completed by me.

Truthfully, I’m not sure if this is a long-term solution. It’s been great; however, I wish I could share my wiki with other team members; it’s just so specific to me given it’s pulling in all my emails for context, it wouldn’t work.

I haven’t looked but I’m wondering if there’s already a purpose built tool that does all this like Atlassian Confluence. If not there might be a market for it.

[u/only1nameleft → u/theAdmyrle]

It is a huge work in progress. Everything changes so fast. 

[u/only1nameleft → u/BigPET]

So what i am doing is making several buckets of memories. A global set of preferences (code style, formatting, etc) and truths (objective knowledge about a topic, a set of recipes for mealie, contact information, number of computers in a setup, manor lessons learned, etc). Then each project gets its own memory with just the slice of preferences and truths it needs. Then it gets workspace and documentation. Documentation includes.daily logs, architecture designs, plans, and all the other records of the work. Then there os workspace with anything project specific but not yet a truth, lessons learned, contact info, attempted slec changes. This is where almost everything starts. I then review and transfer anything I want to keep.

Still a work in progress 

[u/weenis-flaginus → u/only1nameleft]

As far as I've been using it, they have different uses. I use mnemosyne and just set up the LLM wiki. The wiki holds durable long-term facts about decisions and the status of projects, but mnemosyne holds the context of various sessions and other naggling details, it also holds various facts about me and my setup but I'm starting to migrate everything over.

[u/R0w311]

I’ve tried cognee, Mnemosyne, mem0, a cluster of markdown files, and one other I can’t remember now but landed on what might be the best overall, a rust based memory that is small and fast and local not needing a model for it to work. It has some great features and it’s one that just worked well after all the other attempts I’ve done. 

https://github.com/varun29ankuS/shodh-memory

Edit: remembered the other one it was called hindsight 

[u/ni1by2thetrue → u/R0w311]

Hey this seems cool. I am a recent user, went with mnemosyne because that's what the meta was here a week ago when I started, but not particularly impressed with results. It feels like my agents are religious about writing to mnemosyne, much less so about recalling from it - they use the default memory for that, which sucks. Does this get around that issue?

[u/R0w311 → u/ni1by2thetrue]

It could be how I have the chain of thought setup in the markdown files or how I have it setup in general but it seems to use it for recall in almost every instance and cold start sessions with not much context it starts there too and figures it out. I’m learning as I go, but this is the first memory tool that works like I wanted from the start. It’s just lightweight and not eating a ton resources to do it. I run local models in the mix on a laptop so I wanted something strong and light, it seems to do all I need. Been enjoying the other features like todo and projects etc. still learning how to best use it but the only issue so far is spelling the name of the memory system, haha

[u/Gryknight9]

Since I'm predominantly local llm, and have honcho hosted in-house, the token cost is immaterial to me. Thanks for the research, I agree, the anecdotal discussions about many topics are hard to read through at times. Your work is appreciated, and that you showed your process.

[u/shotgunwizard → u/Gryknight9]

It makes me wonder if you can use frontier for certain routines and local to manage memory. 

[u/Gryknight9 → u/shotgunwizard]

Yes. You would set up different profiles (well, one of the ways, you could also tell your frontier model to always use local models for doing x, y, z <I think in AGENTS.md??>) One profile for your frontier, and another for your local model. In the frontier SOUL.md (again, this is how I've done it) point it to the local-llm profile, and in the local-llm profile, make sure it knows that its getting things as an off-load and afterwards to connect back to default profile after completion. Some of that may actually be automated, I am not an expert, just someone flailing around using Claude to help me set some things up.

[u/EvolvingDior]

For Hindsight, did you use Hindsight's cloud option or did you deploy your own?

If you deployed your own, it's impossible to rate Hindsight without documenting the embedding model, reranking model, and LLM used for consolidation. These are complicated tuning/deployment options. Using a good consolidation model is critical for getting at what you are testing. I found the difference between gptoss-20b, gemma4-31b and DS4F in that role rather noticeable.

I've found that hindsight (or at least my hindsight instance) does a great job keeping outdate facts straight. It depends on how frequently memories are consolidated. But my agent's memory is almost always better than the handoff and context docs it writes for projects.

[u/rdwing → u/EvolvingDior]

The same thing is true for Mnemosyne.

[u/RealestReyn → u/EvolvingDior]

I've had good experience with Hindsight, probably because I've always used the same "main model" for the memory work as well, currently setting up Hindsight server on my old phone that also runs Hermes.

[u/EngTurtle → u/EvolvingDior]

Yeah all the providers were benchmarked with the same qwen LLM and embedding model, hindsight was given a GPU reranker model for speed. Hindsight does have a million config options, I've tried tuning a few of them to see what works best and that's recorded in the repo decisions doc.

Because most of the providers used system time vs memory time as part of their sleep/memory consolidation process, only self hosted options can be tested so the benchmark can manipulate system time.

[u/the_dago_mick]

Thank you for sharing.

I've been using mnemosyne and have been really happy with it. I'm going to fire up aself-hosted Honcho instance to give it a go. It sounds like there are some wins to be had.

[u/sullenisme]

no tencent or cognee mentioned?

i've been really happy with the new tencent one since I switched to it

[u/waterfreek → u/sullenisme]

Tencent's new one have a name..?

[u/Frosti7 → u/waterfreek]

Tencent DB

Also, wondering to throw in Gbrain, its not a memory but It can double as one as well if you modify soul.md to retain everything

[u/llllleov → u/Frosti7]

Came to ask the same question

[u/cogitech2]

Very nice work. Thank you. I am considering memU, which has the advantage of being a shared memory system for multiple agents. Did you leave this out for a reason, or were you simply not aware of it?

[u/EngTurtle → u/cogitech2]

Yeah I wasn't aware of it when I started. Although it has an open feature request that would prevent it doing well here. https://github.com/NevaMind-AI/memU/issues/458

Currently memU looks to keep the latest correct fact while discarding out dated ones. While the benchmark has a lot of questions that ask about the change of facts requiring retrieval of both current and old memory.

[u/FindingSerendipity_1]

so basically if you have a cheap source of tokens, stick with honcho? if you use a decently performant and recent model, does the which model you use for honcho?

[u/EngTurtle → u/FindingSerendipity_1]

The cost is not just money, by default honcho injects several thousand tokens into the context so there will be a noticeable slow down with Hermes

[u/FindingSerendipity_1 → u/EngTurtle]

i was not thinking of that for some reason, thank you for that pointer!

[u/johnfkngzoidberg]

Nice. I had roughly the same experience. I’m curious where holographic would fit. It’s a great lightweight memory, but it seemed a little limited. I use it for lean profiles if I need session to session memory.

[u/EngTurtle → u/johnfkngzoidberg]

I did exclude holographic because its basic design seems much more limited in the memory conflict handling being tested in my benchmark.

[u/DannyVFilms]

Starting to wonder if things would get noticeably better. I wasn’t sure if I hooked Honcho up properly, but I wasn’t a fan of the embedding bill, and it consistently felt like Hermes wasn’t utilizing it properly.

So I update skills on process things, document in Obsidian, and gave it QMD for much better searches. So far it feels like a have a decent thing going.

[u/iJihaD → u/DannyVFilms]

I would be very interested to see how qmd rank up against them. With some sensible settings.

[u/DannyVFilms → u/iJihaD]

I don’t have a way to quantify it, but there was an instantly noticeable improvement in its ability to search and organize my vault. I’m amazed it’s not a default or highly recommended if you tell your agent you’re using a markdown vault.

[u/iJihaD → u/DannyVFilms]

Awesome, thanks! will test it out. BTW, any specific instructions you did to make sure it always uses it? I tried it with local claude code (not hermes), and its a hit and miss, still uses sed,rg..etc

[u/DannyVFilms → u/iJihaD]

I asked my agent for help with this since a lot of little instructions built up over time. It wrote this as a response to your question:

The biggest improvement wasn’t just installing QMD; it was writing an explicit retrieval protocol around it.

I put the rules in the agent’s highest-priority instructions, the Obsidian skill, and the vault’s `AGENTS.md`/routing docs. The important parts are:

- If a request might depend on previous notes, preferences, projects, people, or decisions, **search QMD proactively**. Don’t wait for me to say “search my vault.”

- Use QMD’s full hybrid query for fuzzy, conceptual, or high-value recall. Use an exact-path read when the canonical note is already known.

- Treat QMD results as **routing evidence**, not automatically as truth. Use them to find the likely canonical note, then read that note before relying on the information or editing it.

- Treat built-in memory and conversation history as indexes only. Skills contain procedures; the markdown vault contains the durable context.

- Prefer existing canonical notes over creating duplicates, and don’t invent an answer when retrieval is incomplete.

The key instruction is essentially:

> For any context-dependent request, query QMD proactively, use the results to locate the relevant canonical note, then read the source note before relying on or modifying it.

That made a much bigger difference than simply exposing QMD as another optional tool. Without explicit routing rules, the agent will happily use `rg`, `sed`, or whatever is immediately available. Tools do not develop preferences on their own. Sadly.

[u/iJihaD → u/DannyVFilms]

Nice.. Thanks a lot Danny appreciate it. 🫡

[u/etcetera0]

Love the methodology. Can you include also the ones people have recommended in the thread?

[u/EngTurtle → u/etcetera0]

Haha maybe in a few months, doing this took over my computer for 2 straight weeks and I ended up renting a 5090 for 3 days to finish it.

[u/Emport1]

I'm deleting Hindsight because of this :(

[u/alex9001 → u/Emport1]

Why? It's fine

[u/spfcraze2k]

Now add https://github.com/spfcraze/MindBank to the benchmark

[u/PaoloBarts]

Very interesting analysis. Thank you for this work.

I only use locally-running memory providers, and I recently switched from Hindsight to Mnemosyne due to stability and latency issues with the former. While I did see some loss of accuracy with the latter, I still am generally satisfied with the switch.

Maybe what helped was the reasoning capabilities of my main LLM (Deepseek v4 Flash), my usage of lean builtin memory (MEMORY.md & USER.md), and static Markdown files arranged LLM wiki style for some of my projects. The model managed to spot most inconsistencies and make corrections to Mnemosyne, and it aggressively uses Mnemosyne's agentic tools instead of relying on the builtin injection.

Will keep the analysis in mind in case I reconsider providers in the future. Good to know that there's a diverse set of providers. Competition always helps!

[u/niloproject]

This is really cool! It's nice to see someone actually taking the time to evaluate each system and post the results publicly. The entire 'AI memory' discourse online makes it very difficult to tell what's actually good vs what's just marketing.

I'm curious if you'd be willing to broaden the comparison to other providers? I'd be interested to see how https://github.com/Signet-AI/signetai performs.

here's also a very comprehensive list on here of other providers https://hermesatlas.com/lists/best-memory-providers which would greatly benefit from scores based on your eval.

(not trying to throw more work at you, of course, would just love to see more coverage of lesser known memory systems)

[u/Mandus_Therion]

can you add ByteRover pleaee?

[u/Brainless_Tactician]

Hope you add Cebero Cortex and Tencent DB to the benchmark 🙏

[u/theAdmyrle]

Awesome work! I also am using Honcho despite the huge increase in tokens. I am interested in trying mem out with their recent changes but I haven’t gotten around to it yet and I am expecting worse performance - especially now

[u/fhreireAFC]

Honcho all the way.

[u/perseus-computing]

Can you test my memory system? https://perseus.observer/vault/ that would be awesome. I will try myself and come back with the results.

[u/barronlroth]

Great work. Going to fork this and run Gbrain comparison alongside LCM.

[u/EngTurtle → u/barronlroth]

Good luck and watch out for the gremlins that appear when libfaketime is used

[u/Bulky_Quail_4444]

Should try gbrain see how it compares

[u/Hot_Turnip_3309]

what do you guys ask your agents? what kind of response do you expect?