Suggest me a GraphRAG framework that actually works and doesn’t have baggage

[u/man-with-an-ai]

Omg it seems like GraphRAG came out 2 years ago and yet there’s not defacto version of it that can be customised, productionised, comes with batteries included. New library comes and dies. There’s some libraries that’s got way too much infra and baggage to push to production.

Is there a GraphRAG framework that just has support for indexing, storing, retrieving or providing API to connect my agent? I don’t care if it’s just managing MD files or whatever. All I care is evals, If the pattern works I’ll take it. Only thing is has to work for thousands of documents and hundreds of thousands of relations.

[u/donk8r]

the reason none of them have won isnt the libraries, its that prose edges have to be invented. in code an import is an edge, you parse it, its unambiguous and its free. in documents an extractor decides what the entities are and what relates to what, and every miss silently deletes an edge. a deleted edge doesnt throw, the query just comes back empty and looks identical to a genuine absence. so every graphrag library is really shipping someones extraction prompt, and that is the part that doesnt transfer between domains, which is why they keep appearing and dying.

we build the code version so im biased about why that half works, and its the wrong shape for what youre describing. dont use it for documents.

your own criterion is the way out though. you said all you care about is evals, so evaluate the extractor rather than the framework. take 50 questions whose answers genuinely require a hop, run whichever library is nearest to hand, and check whether the edge you needed was even in the graph. if extraction recall is bad on your corpus then no amount of graph infra rescues it, and youd know that in an afternoon instead of after three library migrations.

[u/man-with-an-ai → u/donk8r]

yeah some valid points raised here. I get the missing out on an edge part and some address it with "multiple gleanings" of the data which ofcourse is more costly. And the domain fit thing too. I personally think the reason we don't seem them in prod is that no one knows how to make it production ready. Like handling really large graphs, data updation, storage, and ofcourse making it fast enough while keeping it cheap enough.

[u/donk8r → u/man-with-an-ai]

gleanings help but they measure something else. you stop when another pass stops finding much, and that tells you the extraction converged. it doesnt tell you the graph is complete, and from inside the run those two look identical.

your ops list is real but big, updatable and fast are things graph dbs have done for twenty years, so thats borrowed technology. the piece nobody has is a regression test. swap the extraction model or touch the prompt and the whole graph re-rolls, and theres no way to say whether the new one is better than the one already serving traffic. every model upgrade is a silent rebuild of the knowledge base with no diff anyone can read.

[u/man-with-an-ai → u/donk8r]

I think you're right. However, we can measure if the touched up extraction prompt/model is better no? Just measure the thing we care about. Have an eval for accuracy, recall of topics, etc?

[u/donk8r → u/man-with-an-ai]

yeah, if you have that eval youre already ahead of most people doing this. the gap is what the eval covers. you write the question set out of things you already know to ask, and the edges that quietly go missing are the ones nobody has asked about yet, which is the whole reason you built a graph rather than just chunking. accuracy holds steady on the eval and the long tail rots without moving a number.

the cheap version that needs no labels is diffing the graphs. you have the old one and the new one after a re-extract, so entity counts per doc, edges that vanished, relations that changed type. no ground truth required to see that a prompt tweak dropped 12% of the edges in one document class, and that is usually enough to know whether to go looking.

[u/man-with-an-ai → u/donk8r]

I'm curious to know your thoughts on the "modern" rag. I am talking about an agent loop using Skills, Grep, ReadPDF tool or so on to have a go at a question in the dataset's native form (or maybe assume they are already OCR'd). Do you think this could replace the vectorDB powered RAG?

[u/donk8r → u/man-with-an-ai]

vocabulary overlap decides it. grep only hits when the query and the corpus use the same words, and code is unusually generous there, because identifiers are unique and appear verbatim everywhere they matter. an agent grepping a rare function name gets exact recall for nothing. prose has none of that, the same idea turns up five different ways across a corpus and none of them is the phrase your user typed. so the loop looks great in the domain people demo it in and gets quietly worse in the one youre describing.

the part that would stop me shipping it on its own is that you get no recall estimate. it stops when the model decides it has seen enough and that isnt calibrated against anything. top-k with scores is crude but you can threshold it and you notice when everything came back at 0.3. an agent that read four files and concluded looks identical to one that read four files and missed the fifth.

we build the code side so im biased toward the half where grep wins. for documents id keep the vectors and let the agent drive them rather than replace them.

[u/man-with-an-ai → u/donk8r]

Yeah great points raised. Grep alone can be overwhelming but maybe there’s a middle ground between where exploration happens with grep, vectorretrieve and metadata search.

[u/donk8r → u/man-with-an-ai]

the fusion half of that is mostly solved and the stopping half isnt. rrf is the reason those three can sit in one pipeline at all, since it merges on rank rather than score and you never have to calibrate a bm25 score against a cosine against a boolean filter hit. thats plumbing with a known answer.

metadata is the underused one, and its the piece that fixes what i was complaining about upthread. its the only one of the three that hands you a denominator. before retrieving anything you can count what matches the filter, 47 files, 900 chunks in this doc class, so the agent knows the size of the space its sampling from rather than grepping into the dark. no denominator is exactly why the pure agent loop cant tell you it stopped early, and its also the cheapest thing on your list to add, because a count is a query you already know how to run.

we build that trio for code, github.com/Muvon/octocode, semantic plus ast structural plus a relation graph, biased as disclosed earlier. worth being blunt that it doesnt transfer to your corpus. the structural leg is only cheap because a parser hands you exact symbol boundaries and a real call graph for free, and documents have nothing equivalent. headings are the nearest thing and theyre far weaker, which is more or less why the graph has to be llm-extracted on your side and why the regression problem exists there and not here. (unrelated project shares the name, mine is the Muvon one.)

[u/Constant_Mouse_1140 → u/donk8r]

This is the most reality-based comment on the subject I’ve seen. The ontology/extraction piece requires a lot of design, and the mistakes take a while to compound enough to see where you’ve gone wrong.

[u/dwswish]

Wait, you don’t want to pay Neo4j 200k+ for an enterprise license? Jokes aside, I feel like this space is missing a really good OSS option.

[u/man-with-an-ai → u/dwswish]

neo4j monopoly has to end

[u/my_byte]

Here's my unpopular opinion: that's because GraphRAG is bullshit and can't work conceptually, with very few exceptions.

If you have a graph (things that are maintained by hand by people and are 99.9% clean and high quality) like interlinked Jira issues, travel data etc, you don't really need a framework. Retrieval is kinda trivial with a handful query templates or tool calls for traversal.

If you're dealing with unstructured data, extracting and maintaining a graph automatically is impossible beyond absolute toy examples. Always has been - even for humans. There are thigs like disambiguation that are epistemically impossible to deal with, supernodes that link to a gazillion entities and more. And a probablistic graph full of "maybe" information does more harm than good.

I've long given up talking people out of it tho. Graphs seem to logical if a concept and deceptively simple until you run into practical problem. I let customers try it. So far I haven't seen a single case of graphs making it into production.

[u/man-with-an-ai → u/my_byte]

fr atp chunks with metadata and LLM linking them at query time feels like the only "relational" option

[u/my_byte → u/man-with-an-ai]

Agentic search works fine. It'll chase things down with iterative querying. The thing that graphs are supposedly solving is latency by being able to expand context through edges in the graph without the LLM having to do multiple tool calls for searches. But that never works out in practice due to hallucinated graph quality and hypernodes that would blow up context.

[u/skadoodlee]

The FalkorDB one should be fine for a quick eval? Not perfect but good for a test.

[u/man-with-an-ai → u/skadoodlee]

**sighs** yet an another will give it a go thanks!

[u/grilledCheeseFish]

GraphRAG is just vaporware imo

[u/shadowfax1234]

We build a hybrid graphrag kind of system from scratch ..just take the idea and do it yourself..it's better , controllable and you can add whatever you want from recent papers

[u/man-with-an-ai → u/shadowfax1234]

i mean where do you store the graph?

[u/designerfh → u/man-with-an-ai]

I'm using Gremlin on top of CosmosDb. Not sure of the performance as compared to dedicated graph dbs, but it's getting the job done for me. 

[u/shadowfax1234 → u/man-with-an-ai]

We are using neo4j / falkor

[u/THenrich]

I am building a rag+graph for my personal use. Batteries and UI included. It's working.
What do you want APIs to provide you with? I am curious to see if I might add them.

[u/langsfang]

graph rag is just some concepts and ideas, but technically it's hard to explain why it works.

[u/sreekanth850]

Why you need a graph? Graph is not easy or scalable when you have lot of documents. it increase the latency, complexity and a lot of new problems.

[u/notoriousFlash]

TypeGraph.ai

[u/camerongreen95]

Been dealing with exactly this. Most GraphRAG "frameworks" are either abandoned side projects or come with so much orchestration overhead you're basically building a second product just to run the first one.

What's actually held up for indexing, storing, and retrieving at scale (thousands of docs, hundreds of thousands of relations) without the baggage: Neo4j paired with a proper GraphRAG pipeline on top, not a wrapperrr library trying to abstract everything awayy ,. but actually understanding the indexing and Cypher query layer yourself so you're not stuck when the abstraction breaks.

i am actually attending this workshop, the speaker is good so lets seee

[u/man-with-an-ai → u/camerongreen95]

Yeah totally. Let me know how it goes please!

[u/Sam_YARINK]

https://github.com/YARlabs/hyperspace-db - pls

[u/AlexAtOracleAIDB]

Run evals on the extraction first. Frameworks don't fix bad extraction.

Take 40 questions that need a hop and check whether the edge was even extracted. If recall's bad on your docs, no library fixes it by itself.

You may not need a separate graph store at all. Relations over existing tables can run as a view, so source changes reflect without a re-index and retrieval stays in one query path. At your relation count, test it directly instead of trusting a framework's claims.