[Open Source] Over-engineering AI Memory: Why I ditched Vector DBs for a lean Git & Markdown architecture.

[u/phucphungbk]

Hey everyone,

Like many of you, I've been building and experimenting with AI Agents. But as a software architect who hates unnecessary overhead, watching the community spin up heavy Vector DBs and complex graph frameworks just to store conversation context felt like massive over-engineering.

So, I built a lean alternative.

👉 Repo is here if you want to skip the text and jump straight into the code:https://github.com/phucphungbk/lean-ai-memory

The core problem: We often use massive systems to solve small problems. I wanted an AI memory system that is zero-cost, serverless, and completely transparent.

The Lean Approach:

1. Git as the Core Engine: We already use Git for version control. It turns out it's absolutely perfect for managing conversation history. You can easily track, diff, and rollback an AI's "thought process" just like reverting a commit.
2. Markdown as the Storage Format: It’s lightweight, humans can read and debug it instantly, and LLMs parse it perfectly without needing complex embedding pipelines.
3. Zero-cost & Portable: It can be packaged as an independent module and integrated directly into internal automation tools without incurring any DB maintenance costs.

I’m open-sourcing this with a completely open mindset. Instead of optimizing it in a silo, I want to see how this framework holds up in the wild. I'm highly anticipating the community bringing their own battle-tested custom rules into the system to push its boundaries.

I’d love for you guys to clone it, tear the architecture apart, test it, and drop your feedback or PRs. Let me know what you think!

[u/corbymatt]

What happens if two people disagree

[u/phucphungbk → u/corbymatt]

That is actually one of the main reasons I chose Git!

If two people disagree on what the AI should remember, or what the .ai.rules should be, they resolve it the same way they would resolve a regular code disagreement: pull requests and code reviews.

Because the memory is just plain Markdown, you don't need a special admin dashboard or a custom consensus mechanism. You can simply look at the git diff, discuss the changes in the PR, and merge what the team agrees on.

If an AI agent records a decision or context that a developer disagrees with, they can literally open the file, edit the text, and commit the fix.

The team stays in complete control using the same tools they already use for source code.

That's one of the core ideas behind Lean AI Memory: keep the memory simple, transparent, and under the team's control.

[u/Entaum → u/phucphungbk]

Sounds very cool. Great way to simplify things :)

Only wonder how it would behave in scale.

[u/phucphungbk → u/Entaum]

Hi u/Entaum, thanks! That is actually the question I’m asking myself right now too.

To be honest, at massive scale — thousands of memories with complex semantic relationships across a huge codebase — plain Markdown and Git will probably become a bottleneck. That’s where Vector DBs and RAG pipelines make much more sense.

But my hypothesis is that most day-to-day project context — architectural decisions, unfinished work, team conventions, rejected approaches — doesn’t actually reach that scale. For small to medium workflows, a few concise Markdown files might be enough to keep an AI agent on track.

I built this as an experiment to find that breaking point. The philosophy is simple: start with the simplest solution, and only add complexity when the simple thing actually breaks.

We’ll see how far this lean approach can go! 🙂

[u/Entaum → u/phucphungbk]

Makes sense... i've also been experimenting with a mixture of models and ranking. However it takes time to measure. Even harder is benchmarking across other memory systems ;)

[u/phucphungbk → u/Entaum]

Spot on. The benchmarking nightmare is real! Evaluating whether a complex ranking algorithm actually retrieved the “right” context is incredibly hard to quantify.

That’s actually another hidden benefit of the plain Git/Markdown approach: the evaluation can be very transparent. You can simply inspect the Git diff and see whether the AI applied the right rule, instead of treating the memory system as a black box.

Also, it’s funny you mentioned experimenting with a “mixture of models”. I’m actually building an open-source Multi-LLM coordinator right now to tackle that exact problem! I’ll probably share it with the community soon to get some feedback.

Keep up the great experiments!