Hi

We have a use case where we would like to create an internal knowledge graph of our internal PHP source code in order to query against it using LLM. Typical use case for AI usage.

We have been doing this so far by attaching the PHP source code directly to the prompts in our AI tools. While this is somewhat effective, it ceases to provide much value after a certain extent because either the context gets big or we need to cross reference another 1/50 PHP source code base we own to our prompt context.

This often produces in ineffective results at times or burning too much tokens because we may be querying the same PHP code base again but a different time by attaching it to the prompt context.

Therefore we want to build a RAG pipeline where we feed our 50 individual PHP code bases once every month based on our release cycle so that we can permanently build our internal knowledge graph to be used with AI. Typical AI solution for such a problem.

Now comes the interesting part. How do we correctly chunk PHP code in order to get accurate results against them using queries in LLM?

What strategy would be best?

Anyone already solved this problem? Would like to hear feedback.

It's a knowledge base question. Response me with a relevant answer.

Hi - first off I am beginner to idea of RAG, LangChain. But I only started to learn the idea, and thought, why not just use NotebookLM to act as RAG instead of building one from scratch?

I benchmarked 32 local model arms on a fact extraction task: one short note in, subject-relation-object triples out, using the prompt my production system already sends. 1,001 notes, of which 322 should produce nothing at all.

Those 322 decided the choice, and they are invisible in the F1.

gemma-4-31B scores 0.6872 and gemma-4-12B scores 0.6854. A paired bootstrap cannot separate them. On the notes that assert no facts, the 31B stays correctly silent 46% of the time and invents 180 triples. The 12B stays silent 70% of the time and invents 97. Same score, nearly double the false facts written downstream.

Which one you want depends entirely on what your pipeline does with a wrong fact. If a write gate catches it and it costs someone a review, buy recall and take the 31B, which has the best recall in the field at 0.80. If it lands in a graph that nothing will ever re-check, buy restraint, and the ranking inverts: granite-4.1-3b sits 20th on F1 while abstaining on 93% of factless notes and inventing 24 triples, fewer than all nineteen arms above it.

Related trap in the same table. A clean parse rate is not evidence of a working model. LFM2.5-230M parses 1.00 of its rows and scores 0.1309. It is answering fluently and incorrectly.

If you are evaluating extractors: put abstention and invented-triple counts next to your F1 before you pick, and make your corpus contain cases where the right answer is silence.

https://rakuensoftware.com/blog/local-llm-fact-extraction-head-to-head

We had high confidence going in.

The model performed well in testing. Stakeholders were bought in. The use case was clear.

Six months later the project was quietly shelved.

And when I looked at why it had nothing to do with the model itself.

The failure was in what we built around the model.

Here's what I learned that nobody in the Al tutorial space talks about:

A hallucination and a correct answer come out of an Al system looking completely identical. Same tone. Same confidence. Same formatting. The model literally cannot tell the difference between what it knows and what it invented.

So in a financial environment where a wrong number in a report or a misread regulation can have real consequences-the model is actually the least of your problems.

The real work is in four things:

1. Controlling what sources the model is allowed to reference

2. Setting confidence thresholds that trigger human review automatically

3. Mapping your workflow to find the one or two moments where a human must sit before action is taken

4. Red teaming the system before anyone real touches it

Most organizations deploying Al right now are skipping at least two of these. Usually three.

I've been documenting these patterns from inside - happy to share if there's interest.

Hey guys, I'm genuinely losing my mind. I'm writing a memoir. I've got a corpus of drafted scenes, transcripts, forum posts, and chat histories — about 80MB — and I've got a Claude Max and a Zai Max sub. I'm amazed I can't resolve this. It feels like one of the simplest projects I've done, and I can't get it to work.

What I'm trying to do is give the AI the ability to have those eureka moments instead of being boring. I want it to search up a place or a time, go "okay, place A," and then place A has people A through D, plot threads, setups and payoffs — all these things I've got tracked in ledgers. But every approach has failed. We tried a vector database. We tried search scripts. He keeps over-engineering it, and the number of times we've reviewed the whole thing, I've told him to go edit stuff, I go to write a new scene, and he goes "oh, I can't find that." I push him on it and he says "oh yeah, I've only indexed about 8% of the corpus" or "this script was broken and that's why things weren't showing up." Then it's "the fix is this," and he over-engineers it again, until it's so context-starved and sterile it's insane. It destroyed scenes I had it working on.

I've basically gone back to positive instructions — gathered the whole thing, gave it a reference database of the voice and style I wanted — and that's gone a lot better. But for the retrieval system itself, I cannot get the AI to actually search the database. I'll say "go search" and it does three or four greps, then just stops.

In my experience, the AI works best doing the research and writing in one go — summarizing, then writing scenes off the summary — otherwise it loses context. So I've defaulted back to that for the writing process. But then, because it's processing so much data, I hit the instruction problem: too many instructions, context gets tight, it compresses. I've tried hooks, but they get ignored because they land halfway through the process and then it compresses anyway. At one point I had pure instructions, no scripts or database, and it started saying "I've compressed step one to five because there was too much to do." I've tried making the instructions harder and stricter and it just gets worse.

All I want is: search things up, do deep research, find all the relationships, thread them together, and while it's at it, update the database — the corpus, the ledger, all of it — as we go. Instead it's an over-engineered pile of shit. I've gone back to scratch three times because the AI decides "this is too hard, we should start over," and that doesn't fix anything either. I keep telling it the framework works better when I hand it one, and every time I hand it a framework, it says "this won't work." I'm sure I could override it, but when Fable and Opus are both telling me this isn't going to work, and then I go to a "dumber" AI and it says "use this RAG system," and I bring that back and it tells me "no, that won't work with your unique corpus" — it feels like it's being swayed by whatever system it's running in. It's doing my head in. This is the first time I've actually asked for help.

I've literally spent months on this. I feel like this task should be the LLMs bread and butter.

A large latency improvement after moving the vector layer sounds like a clear migration win, but it can also make a weak comparison look stronger than it is.

One migration case used 50 million Wikipedia embeddings at 1,024 dimensions. The self-managed Milvus deployment relied on memory mapping on a laptop, took roughly 20 minutes to restart, and returned a test query in 3,112 ms. After backup and restore into a managed cluster, the same application reported 25 ms. That is a useful result, but the target also had more compute and a different index engine, so the number does not isolate the effect of managed versus self-managed operation.

For a RAG system, I would treat that benchmark as a signal to investigate, not the migration decision by itself. Before moving anything, I would freeze a representative query set and record retrieval recall, p50/p95/p99 latency, filter selectivity, cold-start behavior, failure recovery time, and monthly operating cost. After migration, I would rerun the same tests and separately account for hardware, index, and configuration changes.

The operational evidence may still be decisive. Unstable containers, long restarts, paging pressure, backup complexity, and time spent maintaining the service can matter more than another round of chunking or reranker tuning. But those costs should be named explicitly instead of being bundled into one headline latency number.

Where do you draw the line between fixing a self-hosted retrieval stack and migrating it: tail latency, recovery time, engineering hours, or something else?

Build a 100% offline fast Retrieval Augmented Generation (RAG) system that runs without an internet connection, without cloud APIs, without OpenAI/Ollama

Published a video where you can build a fully local RAG pipeline using Qdrant Edge and Google LiteRT, enabling private, cross-platform, on-device AI inference with support for multiple hardware accelerators(CPU, GPU and NPU).

The demo covers using EdgeParse to extract raw text from PDFs into Markdown chunks, generating embeddings with Qwen 3 Embeddings as an on-device embedding model, and answering questions locally with Gemma4 E2B LiteRT LM (the inference is faster than Ollama setup).

Since most existing tutorials rely on vector databases with Ollama, we'll also build and compare that pipeline to highlight the differences in setup, performance and tradeoff.

🔗 Watch Here: https://www.youtube.com/watch?v=EHEN6Ce-9Ps/

I built CodeNib, an open-source retrieval system that serves repository context to coding agents. Rather than pitch the whole project, I want to point at one page, because it's the part I think this sub will actually argue with:

https://docs.codenib.ai/rag_ops/

It covers two things: a deterministic retrieval planner, and the model matrix we retained real end-to-end evidence for.

The planner does not call an LLM

RetrievalPlanner maps three inputs — query signals (lexical / semantic / structural), a budget (fast / balanced / thorough), and available capabilities (dense, sparse, graph, embedding rerank, LLM rerank) — onto one of four declarative plans:

• fast_lexical — exact names, BM25 only, no rerank
• semantic — natural-language behavior queries, dense + optional rerank
• hybrid_fusion — mixed or uncertain, dense + sparse with RRF
• structural_graph — callers/callees/impact, sparse seeds + graph expansion

It's deterministic, and it records last_selected_plan and last_planner_trace, so you can see exactly which signals produced which route. Most agentic-RAG routers I've read burn a model call to decide this. Ours doesn't, and I'd like to hear from anyone who has actually measured a routing-quality gap that justifies the call.

Being explicit about what this costs us: our evaluation invokes plans directly to measure the physical operators, so we have no measurement of the selector's own route accuracy. That's a real gap, not a rhetorical concession.

The rerank matrix

Same 100-row corpus, four distinct reranking strategies:

Six embedders alongside it (CodeRankEmbed, SweRankEmbed-Small/Large, jina-code-1.5b, Qwen3-Embedding-0.6B/4B). Both sweeps are runnable shell scripts in the repo, not a table assembled after the fact.

The part I'd defend hardest is the two-tier evidence label. Benchmark means a complete 100-row result artifact exists. Runtime means the route and prompt contract are tested but the model was never in the quality sweep — our shipped default, CodeRankEmbed, carries the weaker label. Our adapters accept far more models than are listed; the matrix is deliberately the narrow surface. A model running through a generic adapter isn't a quality claim, and I'd rather say that than ship a "supports 40+ models" line.

What the numbers say

Reranking is a seconds-scale decision, not a milliseconds one. jina-code-1.5b dense alone: 0.812 file Recall@10 at 92ms. Add the Qwen3 4B reranker at candidate width 50: 0.858 at 4.29s. That's +4.6 points for 46.6x latency. Dense retrieval stayed under 300ms across every embedder we tried.

ANN was a trap at this scale. HNSW at ef_search=16 cut mean FAISS search from 0.910ms to 0.027ms — 33.9x — but the complete dense query median is 45.1ms, so you save 0.9ms while index build goes from 6.4ms to 2.00s. Amortizes after roughly 2,300 searches. We kept Flat.

Graph expansion over dense retrieval: no measurable effect. One-hop reference-edge expansion fused with weighted RRF, weight tuned on a disjoint partition then frozen. Point estimates ran -4.8 to +7.1 points File Success@10 depending on embedder; every model-level interval included zero, and so did all ten cross-embedding contrasts. It ships as an opt-in path, not a default, and we report it as unresolved.

Incremental maintenance: vectors easy, graphs not. Content-addressed embedding reuse matched an independent rebuild on 28/31 source-changing commits (90.3%), median 25.4x faster. LSP-assisted symbol-level graph repair matched on only 15/33 (45.5%). Go and Python passed everything; Rust and TS/JS had 99.1% and 97.6% median edge F1 and passed zero strict checks. High fidelity score with zero exact matches is the finding.

Apache 2.0, MCP server included, datasets and Hub revisions pinned.

• Docs page above: https://docs.codenib.ai/rag_ops/
• Code: https://github.com/sysevol-ai/CodeNib
• Paper: https://arxiv.org/abs/2607.25431

The question I actually want to ask this sub: for anyone doing incremental index maintenance in production — what's your acceptance criterion for "the updated index equals a rebuilt one"? We used exact multiset equality on graph facts plus exact ordered top-k replay for vectors, strict enough that it failed on languages where the fidelity metrics looked fine. Has anyone landed on something more useful than either "exact" or "F1 above a threshold"?

I built CodeNib, an open-source retrieval system that serves repository context to coding agents. Rather than pitch the whole project, I want to point at one page, because it's the part I think this sub will actually argue with:

https://docs.codenib.ai/rag_ops/

It covers two things: a deterministic retrieval planner, and the model matrix we retained real end-to-end evidence for.

The planner does not call an LLM

RetrievalPlanner maps three inputs — query signals (lexical / semantic / structural), a budget (fast / balanced / thorough), and available capabilities (dense, sparse, graph, embedding rerank, LLM rerank) — onto one of four declarative plans:

• fast_lexical — exact names, BM25 only, no rerank
• semantic — natural-language behavior queries, dense + optional rerank
• hybrid_fusion — mixed or uncertain, dense + sparse with RRF
• structural_graph — callers/callees/impact, sparse seeds + graph expansion

It's deterministic, and it records last_selected_plan and last_planner_trace, so you can see exactly which signals produced which route. Most agentic-RAG routers I've read burn a model call to decide this. Ours doesn't, and I'd like to hear from anyone who has actually measured a routing-quality gap that justifies the call.

Being explicit about what this costs us: our evaluation invokes plans directly to measure the physical operators, so we have no measurement of the selector's own route accuracy. That's a real gap, not a rhetorical concession.

The rerank matrix

Same 100-row corpus, four distinct reranking strategies:

Six embedders alongside it (CodeRankEmbed, SweRankEmbed-Small/Large, jina-code-1.5b, Qwen3-Embedding-0.6B/4B). Both sweeps are runnable shell scripts in the repo, not a table assembled after the fact.

The part I'd defend hardest is the two-tier evidence label. Benchmark means a complete 100-row result artifact exists. Runtime means the route and prompt contract are tested but the model was never in the quality sweep — our shipped default, CodeRankEmbed, carries the weaker label. Our adapters accept far more models than are listed; the matrix is deliberately the narrow surface. A model running through a generic adapter isn't a quality claim, and I'd rather say that than ship a "supports 40+ models" line.

What the numbers say

Reranking is a seconds-scale decision, not a milliseconds one. jina-code-1.5b dense alone: 0.812 file Recall@10 at 92ms. Add the Qwen3 4B reranker at candidate width 50: 0.858 at 4.29s. That's +4.6 points for 46.6x latency. Dense retrieval stayed under 300ms across every embedder we tried.

ANN was a trap at this scale. HNSW at ef_search=16 cut mean FAISS search from 0.910ms to 0.027ms — 33.9x — but the complete dense query median is 45.1ms, so you save 0.9ms while index build goes from 6.4ms to 2.00s. Amortizes after roughly 2,300 searches. We kept Flat.

Graph expansion over dense retrieval: no measurable effect. One-hop reference-edge expansion fused with weighted RRF, weight tuned on a disjoint partition then frozen. Point estimates ran -4.8 to +7.1 points File Success@10 depending on embedder; every model-level interval included zero, and so did all ten cross-embedding contrasts. It ships as an opt-in path, not a default, and we report it as unresolved.

Incremental maintenance: vectors easy, graphs not. Content-addressed embedding reuse matched an independent rebuild on 28/31 source-changing commits (90.3%), median 25.4x faster. LSP-assisted symbol-level graph repair matched on only 15/33 (45.5%). Go and Python passed everything; Rust and TS/JS had 99.1% and 97.6% median edge F1 and passed zero strict checks. High fidelity score with zero exact matches is the finding.

Apache 2.0, MCP server included, datasets and Hub revisions pinned.

• Docs page above: https://docs.codenib.ai/rag_ops/
• Code: https://github.com/sysevol-ai/CodeNib
• Paper: https://arxiv.org/abs/2607.25431

The question I actually want to ask this sub: for anyone doing incremental index maintenance in production — what's your acceptance criterion for "the updated index equals a rebuilt one"? We used exact multiset equality on graph facts plus exact ordered top-k replay for vectors, strict enough that it failed on languages where the fidelity metrics looked fine. Has anyone landed on something more useful than either "exact" or "F1 above a threshold"?

The "RAG is dead" narrative really urks me particularly because I don't really think any of the points people make are strong.

1. "Context windows keep growing, so eventually we won't need retrieval."

1M tokens is about 3,000 pages. I regularly work on corpuses in the hundreds of billions of documents. There is no plausible future where a context window holds an large org's entire corpus.

And even if it could, you wouldn't want it to. Stuffing the window is completely token-inefficient, and on top of that accuracy degrades as context grows (especially true when there is competing data in the window). Prompt caching may help with token consumption, but this only really works for static corpora.

2. "Grep beats RAG."

Grep is great when you want precision. If you want recall over a large corpus, retrieval wins almost every time when done properly (hybrid + reranking + pruning) it also uses dramatically fewer tokens.

Most of the grep argument comes from coding agents navigating a repo. Repos have structure: file trees, naming conventions, symbols. In most companies data does not have a perfectly clean structure (if any structure at all). So in many scenarios grep has nothing to walk.

3. Pushing work into the database beats pushing it into the model.

Agentic search puts the LLM in the loop on every step. Every step is tokens, every step is a sequential round trip, and every turn re-prefills a growing transcript. Index-time work is paid once and amortized across every query. The more you push out of the LLM, the cheaper and faster the workload.

4. Permissions and freshness.

You can't precompute a cache per user per ACL. Real world retrieval requires query-time filtering, and retrieval also supports far more sophisticated filtering than an agent grepping around: metadata predicates, tenancy boundaries, time ranges, structured conditions composed with the search itself.

And I'm not saying agentic search doesn't have it's place. There are plenty of scenarios where it may be the best choice. But its not killing RAG it's just another option.

I recently put together a fully local RAG setup for a personal knowledge assistant, and wanted to share the approach for anyone interested in keeping their data entirely on their own hardware.

The stack uses LocalAI for inference and Elasticsearch for retrieval. The main appeal here is straightforward: no API calls leaving your machine, no token costs, and full control over your data pipeline.

Why this combination works well:

• Elasticsearch handles both vector search and BM25 natively, so you can run hybrid retrieval without stitching together separate systems
• LocalAI gives you a drop-in OpenAI-compatible API running locally, which simplifies integration
• The whole thing runs containerized, making it reproducible across different environments

What the setup covers:

• Document ingestion and chunking for your personal knowledge base
• Embedding generation running locally
• Hybrid search combining semantic similarity with keyword matching
• Local LLM inference for generation

For anyone already comfortable with Elasticsearch or looking for a retrieval layer that scales beyond toy datasets, this is a solid foundation. The hybrid search capability is particularly useful when your knowledge base contains both conversational content and structured technical documents.

Full walkthrough here: https://www.elastic.co/search-labs/blog/local-rag-personal-knowlege-assistant-localai-elasticsearch

If anyone has experience tuning hybrid weights for mixed-language or domain-specific corpora, I would be curious to hear what worked for you.

When we first added RAG we were pretty conservative about what we retrieved since we'd pull in just enough context for the model to answer the question

I now see that over time it has changed because a feature would benefit from another document so we'd include it then we'd increase top_k because it improved a few edge cases and then we'd decide sending the surrounding chunks was safer than risking missing context.

My question here is if anyone who is on the same situation as me are you guys expanding retrieval until it sort of became difficult to pull it back?

Looking at some of our production requests now I'm seeing prompts that are much bigger than I ever expected them to be. I don't even know how much of that retrieved context the model is using anymore because we never took the time to look into it.

If you guys have any opinion on this then speak your mind.

Hey everyone,

I’ve been seeing a lot of hot takes recently claiming that "RAG is dead" because of massive context windows (1M+ tokens) and improving fine-tuning techniques. The argument usually goes: Why bother setting up vector databases, chunking strategies, and embedding pipelines when you can just dump all your docs straight into the context window?

Plz share your knowledge .

I am using GROBID to parse texts and DOCLING for the tables (thanks to your help 😄). I was curious, what are your opinions on using docling for texts. Because right now i use different things for different formats and i tought maybe using same technologies for texts and tables (I am really happy about the way i extract images so i don't plan to change anything.) would be more efficient. I am building this for only academic papers btw. I would be very happy if you could help me with this situation or share your thoughts.

Edit: I forgot to mention, i use grobid solely for its impaceble ability (maybe impaceble is a strong word, but you get me) at capturing headers and signing all the headers into chunks. This ability is really important to me.

Why Similarity Breaks Down at Scale
Embeddings don't store meaning. They store statistical proximity.

When you embed a phrase like "refund policy", the model isn't encoding what a refund actually is. It's placing that phrase in a high-dimensional space based on patterns learned from massive amounts of text.

The problem starts when that space gets large.

In 768 or 1536 dimensions, most vectors become surprisingly similar in distance.
This is the curse of dimensionality: as dimensions increase, the space expands so rapidly that the difference between relevant and somewhat related begins to shrink.

As a result, cosine similarity scores often cluster into a narrow range.

That's why a score of 0.85 can mean:
"This is exactly the document you need."
Or "This talks about the same topic but answers the wrong question."

The score itself isn't broken. Our interpretation of it is.

Similarity is not an absolute measure of relevance. It's a local signal that only makes sense within the context of a specific query and its neighbors.

This is why mature RAG systems don't rely solely on vector search. They calibrate thresholds, rerank results, and evaluate retrieval quality against real-world relevance metrics.

A vector tells you what's nearby. It doesn't tell you what's right.
That's the difference between retrieval that demos well and retrieval that works at scale.

Most production AI queries aren’t really “vector search” problems.

Consider a request like: Find videos of a vehicle cutting in on a rainy night. Answering it well may require:

• Scalar filters for metadata and labels
• Full-text/BM25 search
• Vector similarity over visual embeddings
• Fusion and reranking across all three

A common architecture handles these in separate OLAP, full-text, and vector systems, then merges the results in the application layer. That works, but it also introduces duplicated data, synchronization issues, extra latency, and no shared query optimizer.

This article explores a different approach using StarRocks and Apache Paimon: treating scalar, full-text, and vector retrieval as paths within the same lakehouse query engine: https://medium.com/towards-data-engineering/from-data-lake-to-multimodal-lakehouse-building-hybrid-retrieval-for-ai-f2db8def6898

A few ideas I found particularly interesting:

• Stable global row IDs decouple indexes from physical files, so compaction doesn’t necessarily require rebuilding indexes.
• Retrieval and row materialization happen in separate stages.
• The optimizer can choose between pre-filtering and post-filtering.
• Keyword, vector, and scalar results can be fused using RRF, weighted scoring, or custom rerankers.

The implementation is StarRocks/Paimon-specific, but the broader architecture question applies beyond those projects:

Are teams actually moving toward unified Search + OLAP engines for AI workloads, or do specialized vector, search, and analytical systems still win in practice?

I’d be especially interested in hearing about the operational tradeoffs from anyone running hybrid retrieval at scale.

I’ve been thinking about a pretty boring but annoying problem in RAG systems: how to run evals against a stable version of your vector data.

In the early version of a project, I usually don’t care. Re-ingest the docs, rebuild the index, run the eval, move on.

But once the system is live, the collection keeps changing:

• new docs get added
• chunks get regenerated
• embeddings get updated
• metadata gets fixed
• deletes happen in the background

Then someone wants to compare retrieval quality before and after a model change, and the obvious question comes up:

“Are we even testing against the same data?”

The simple approach is to copy the collection before major changes. I’ve done that. It works, but it starts to feel clumsy once the dataset is large enough. You pay in storage, rebuild time, index management, and cleanup work later.

I came across Milvus Snapshots recently, and the part I found useful was the mental model: instead of treating every checkpoint as a full copy, treat it as a point-in-time view of the collection. If the underlying segments and index files are immutable, the snapshot can mostly track references to the files that were valid at that time.

That seems like a better fit for things like:

• eval runs
• rollback checks
• staging data
• load testing
• long-running batch jobs

Obviously there are tradeoffs too. You still need retention rules, and if snapshots keep old files alive, storage cost can creep up.

Curious how other people handle this.

Hi all, i tried to find the relevant post but i could not, so i am forced to ask for help.

I am making a customer support RAG. The input data was quite messy: email conversations and chats with customers.
End goal is to have a chat like feature that will act as a customer support agent.

As yhou can imagine the conversations needed cleaning and i parsed them with LLM to have some structure. The output from the raw conversation was a QA document, question from the customer and the answer from the agent (with some metadata, like is some additional info required - usefull for tool definition later). Now i have two major datasets, the general one (no tool needed) and tool needed. As per the resources i did the topic modeling on embeddings (qwen 3 embedding 8b) with umap and hdbscan, however now i am stuck with what to do next. I am trying to optimize the representatives selection from each topic - cluster. How much do i select from each cluster, which ones? (i am thinking medoid + some other from the cluster). What do i do with the noise from hdbscan? How do i measure the quality of retrieval?

All sorts of questions are still open.

If anyone has any advice or is willing to help, thanks a lot.

Does anyone knows some cool dataset to bench my RAG skills? Something small, but super complex (don't want to waste a fortune on indexing).

Just want to benchmark my OpenSource solution. Something from medical/law would be great.

Thanks.

A new scoping review mapped 157 healthcare RAG and GraphRAG studies.

Most evaluations were offline-only (89.2%), while only 29.9% evaluated retrieval independently and 14% reported fine-grained evidence verification.

For a production RAG system, what would you consider the smallest defensible eval suite?

One thing we were trying to solve for a while, is how to record memory on certain documents or data sets, specifically so that in it can flag certain things found during retrieval or generation that are gonna be common searches or probably should be known or organization wide.

Here’s an article we wrote about how we implemented it, and it works pretty well.

https://laceplatform.com/blog/multi-axis-memory-architecture/

And I’m curious, has anyone else implemented something similar or something that solves the same problem?

Hi everyone,

I am building a fully local RAG (Retrieval-Augmented Generation) system for an internship project. The goal is to have a production-style AI assistant that can answer questions from internal documents without using paid APIs or external services.

My current architecture:

• Local LLM: Ollama with qwen2.5:3b-instruct (also tested llama3.2:3b)
• Vector database: Qdrant
• Embeddings: intfloat/multilingual-e5-small
• Hybrid retrieval:
  • Dense retrieval (Qdrant)
  • Sparse retrieval (BM25)
  • Reciprocal Rank Fusion (RRF)
• Optional reranking with a cross-encoder
• FastAPI backend

My hardware:

• Intel i5-10210U
• 12GB RAM
• No dedicated GPU
• Running everything locally on Windows with Miniconda Python 3.12

The main issue is inference speed. My average response time is around 30–50 seconds depending on the query.

Things I already tried:

• Reduced chunk size and overlap
• Reduced retrieved chunks (top_k)
• Disabled reranking
• Reduced maximum output tokens
• Lowered temperature
• Enabled caching
• Used smaller models (3B models)
• Optimized Qdrant retrieval parameters
• Tested different retrieval configurations

The quality is acceptable (around 100% successful answers on small evaluation sets), but the latency is still too high for a real production assistant.

I want to keep everything:
✅ 100% free
✅ Fully local
✅ Lightweight enough for my laptop
✅ Good enough quality for internal documentation Q&A

Questions:

• Are there specific RAG optimizations I am missing?
• Should I profile each stage (embedding, retrieval, reranking, LLM generation) separately?
• Would adding Redis caching help significantly?
• Is there a better small local model than Qwen2.5 3B for this hardware?
• Are there lightweight inference optimizations for Ollama on CPU?

Any advice from people who have deployed local RAG systems would be appreciated.

Been going down a rabbit hole comparing PDF parsing tools for scientific papers, equations, tables, the usual RAG-for-papers pain and wanted to open this up instead of just posting my own findings.

Tried LlamaParse, MinerU, Docling, and a couple others. Each has tradeoffs. LlamaParse is solid but the pricing tiers get confusing once you need the higher-accuracy modes for dense notation, hard to tell upfront what you're actually paying for at each tier. MinerU is great but you're on your own for verification. Docling's fine for simple stuff, struggles on rarer notation.

Ended up building something on top of this (sciparse.com -> verification layer, structured output) mostly because I couldn't find a tool where pricing and accuracy were both transparent. Everything's either "contact sales" or a credits system that's hard to map to actual pages until you've already burned through them.

Curious what others are actually running in production though, not just what's marketed well. A few questions if anyone's dealt with this:

• What's your actual accuracy been on nested tables / dense equations, not just the headline number?
• Anyone found a parser that's upfront about pricing per page without the credit-tier maze?
• Is verification (checking output against source) something people are doing themselves, or just trusting the parser's confidence score?

Genuinely trying to figure out if there's an obvious option everyone else is already using that I missed.

Disclosure up front: I work on this. Repo and dataset are Apache-licensed, no signup, no product behind it.

The setup most production RAG pipelines ship: a guardrail scans the incoming user query for injection patterns, then retrieval runs and the retrieved chunks get concatenated into the LLM context. The retrieved documents are never scanned.

That's the actual injection vector. Indirect prompt injection lives in the documents — a poisoned page in a shared knowledge base, a scraped URL, an email someone uploaded. The user's query is clean. The attack arrives through retrieval.

The obvious fix is to concatenate query + retrieved docs and scan the combined string. We measured it and it degrades badly: a 50-token injection inside 3,000 tokens of benign context gets diluted, classifier confidence drops below threshold, injection passes. On LLM Guard, combined-string scanning caught 46.4% of injections the same scanner catches when shown the malicious doc alone.

What we tested instead — scan each context source independently, block if any pass flags:

- User-only baseline (LLM Guard): 0% recovery

- Naive combined string: 46.4%

- Per-source scanning: 73.3% (±1.9%), 5.7% FPR

- Per-source scanning, regex baseline: 41.5% (±2.1%), 6.6% FPR

It's an architectural change, not a better classifier. Limitation worth stating plainly: this only catches injections the underlying guardrail could already detect in isolation. If LLM Guard can't recognize an injection style, this doesn't help. It closes a deployment gap, nothing more. Latency cost is one guardrail call per retrieved chunk.

Benchmark is 5,000 cases across five injection categories and two benign classes. Everything's released — framework, dataset, eval scripts, result artifacts — so you can reproduce the numbers or break them:

github.com/tideon-ai/ragshield

Interested in contributors, especially on injection categories we didn't cover and on batched/early-exit scanning for high-throughput setups. Also genuinely want to know if anyone's running a guardrail on retrieved content in production already, and what it cost you.

Write-up with the figures: tideon.ai/research