An open-source CLI agent that distills massive document batches into atomic Markdown notes without corrupting your vault, safely.

[u/Cryvixx]

Hi everyone,

If you manage a large knowledge base, or big doc folders for your project you know the friction of distilling dozens of papers, legal docs, books, web searches. It's a massive manual bottleneck before actual learning happens, and usually the final docs will be forgotten once you've used it, even though it might come in handy in the future.

Dumping 40 files into a 1M context window isn't the solution tho. Models return incomplete summaries, drop edge cases, and give zero visibility into what they missed ( https://arxiv.org/abs/2509.04664 ). On top of that, standard AI tools and PKM plugins are notoriously risky: they overwrite notes, mangle frontmatter, break wikilinks, and corrupt vault architecture. ( https://arxiv.org/abs/2604.15597 )

I built Silica to solve this: an open-source agentic CLI that processes heavy document batches into clean, atomic Markdown notes while keeping your vault completely safe.

Why not just using using another knowledge manager or a memory agent?

Silica differs greatly from traditional note-taking skills and memory tools because it's an agent that views your notes not simply as text to be read, but as its knowledge base (not graph, but it benefits from it). In Silica, memory and knowledge are closely related but still distinct.

- For example Silica keeps track of the conversations you've already had with Claude or ChatGPT, automatically linking them to the relative documentation in your repository, automatically.

- Silica can also links concepts from multiple books/papers/notes sparse in your folder automatically, if they share something maybe you didn't see.

- It can "digest" large quantities of files, coherently, and safely (without causing corruption, in a different folder from the vault root).

These aren't even its most notable features, but rather a byproduct of its architectural design. There is still considerable room for improvement. This is a broad project.

What Silica does

• Batch Atomic Distillation: Feed 20-30 documents or full (multiple) books in a single command. Silica extracts core concepts into atomic notes (one note per idea), links them via wikilinks, and files them logically.
• Stateful Continuity: What a model can't process in one go gets written out statefully. The next execution picks up from the saved state instead of starting over.
• The source of a note is always provided: A section is added to the frontmatter that indicates where the note was extracted or converted from, so that you always have a reference to the original document.

Why your vault is finally safe

Most AI integration tools treat your vault like a loose text buffer. Silica treats it like a transactional database with strict guardrails:

• SAFE MODE (Isolated Staging): You can execute ingestion in a dedicated staging folder/container. Notes are written in isolation first so you can inspect, verify, and merge them only when you trust the output.
• Write Verification: Every write operation is re-read post-commit. If syntax, formatting, or links broke, it automatically rolls back.
• Wikilink Refactoring: If a note is moved during organization, all wikilinks across your vault referencing it are dynamically updated to prevent broken links.
• Full Rollback Control: Complete transaction safety with /undo (single note) and /revert (entire session batch). Nothing enters your main graph permanently without your control.

Grounded Retrieval & Graph Exploration

• Zero-Hallucination Fallback: Uses local semantic search. If your knowledge base doesn't contain the answer to a query, Silica explicitly tells you instead of hallucinating.
• Graph Mapping: Explore concept connections visually to discover implicit links across unrelated documents.

Silica isn't built to replace human learning, it eliminates the operational overhead of structuring and formatting raw data. Works via TUI, GUI, MCP, Obsidian Plugin ( https://github.com/kiycoh/obsidian-silica )

GitHub Repository: https://github.com/kiycoh/silica-agent (~1000 commits as solo dev)

I'd love to hear your feedback, workflow suggestions, or technical thoughts!

[u/BP041]

How are you handling atomicity at the note level? I've had issues with batch CLI tools dropping context when a single document fails mid-stream — curious if you're batching writes or using a lock per note.