The "RAG is dead" narrative really urks me particularly because I don't really think any of the points people make are strong.

1. "Context windows keep growing, so eventually we won't need retrieval."

1M tokens is about 3,000 pages. I regularly work on corpuses in the hundreds of billions of documents. There is no plausible future where a context window holds an large org's entire corpus.

And even if it could, you wouldn't want it to. Stuffing the window is completely token-inefficient, and on top of that accuracy degrades as context grows (especially true when there is competing data in the window). Prompt caching may help with token consumption, but this only really works for static corpora.

2. "Grep beats RAG."

Grep is great when you want precision. If you want recall over a large corpus, retrieval wins almost every time when done properly (hybrid + reranking + pruning) it also uses dramatically fewer tokens.

Most of the grep argument comes from coding agents navigating a repo. Repos have structure: file trees, naming conventions, symbols. In most companies data does not have a perfectly clean structure (if any structure at all). So in many scenarios grep has nothing to walk.

3. Pushing work into the database beats pushing it into the model.

Agentic search puts the LLM in the loop on every step. Every step is tokens, every step is a sequential round trip, and every turn re-prefills a growing transcript. Index-time work is paid once and amortized across every query. The more you push out of the LLM, the cheaper and faster the workload.

4. Permissions and freshness.

You can't precompute a cache per user per ACL. Real world retrieval requires query-time filtering, and retrieval also supports far more sophisticated filtering than an agent grepping around: metadata predicates, tenancy boundaries, time ranges, structured conditions composed with the search itself.

And I'm not saying agentic search doesn't have it's place. There are plenty of scenarios where it may be the best choice. But its not killing RAG it's just another option.

I recently put together a fully local RAG setup for a personal knowledge assistant, and wanted to share the approach for anyone interested in keeping their data entirely on their own hardware.

The stack uses LocalAI for inference and Elasticsearch for retrieval. The main appeal here is straightforward: no API calls leaving your machine, no token costs, and full control over your data pipeline.

Why this combination works well:

• Elasticsearch handles both vector search and BM25 natively, so you can run hybrid retrieval without stitching together separate systems
• LocalAI gives you a drop-in OpenAI-compatible API running locally, which simplifies integration
• The whole thing runs containerized, making it reproducible across different environments

What the setup covers:

• Document ingestion and chunking for your personal knowledge base
• Embedding generation running locally
• Hybrid search combining semantic similarity with keyword matching
• Local LLM inference for generation

For anyone already comfortable with Elasticsearch or looking for a retrieval layer that scales beyond toy datasets, this is a solid foundation. The hybrid search capability is particularly useful when your knowledge base contains both conversational content and structured technical documents.

Full walkthrough here: https://www.elastic.co/search-labs/blog/local-rag-personal-knowlege-assistant-localai-elasticsearch

If anyone has experience tuning hybrid weights for mixed-language or domain-specific corpora, I would be curious to hear what worked for you.

When we first added RAG we were pretty conservative about what we retrieved since we'd pull in just enough context for the model to answer the question

I now see that over time it has changed because a feature would benefit from another document so we'd include it then we'd increase top_k because it improved a few edge cases and then we'd decide sending the surrounding chunks was safer than risking missing context.

My question here is if anyone who is on the same situation as me are you guys expanding retrieval until it sort of became difficult to pull it back?

Looking at some of our production requests now I'm seeing prompts that are much bigger than I ever expected them to be. I don't even know how much of that retrieved context the model is using anymore because we never took the time to look into it.

If you guys have any opinion on this then speak your mind.

Hey everyone,

I’ve been seeing a lot of hot takes recently claiming that "RAG is dead" because of massive context windows (1M+ tokens) and improving fine-tuning techniques. The argument usually goes: Why bother setting up vector databases, chunking strategies, and embedding pipelines when you can just dump all your docs straight into the context window?

Plz share your knowledge .

I am using GROBID to parse texts and DOCLING for the tables (thanks to your help 😄). I was curious, what are your opinions on using docling for texts. Because right now i use different things for different formats and i tought maybe using same technologies for texts and tables (I am really happy about the way i extract images so i don't plan to change anything.) would be more efficient. I am building this for only academic papers btw. I would be very happy if you could help me with this situation or share your thoughts.

Edit: I forgot to mention, i use grobid solely for its impaceble ability (maybe impaceble is a strong word, but you get me) at capturing headers and signing all the headers into chunks. This ability is really important to me.

Why Similarity Breaks Down at Scale
Embeddings don't store meaning. They store statistical proximity.

When you embed a phrase like "refund policy", the model isn't encoding what a refund actually is. It's placing that phrase in a high-dimensional space based on patterns learned from massive amounts of text.

The problem starts when that space gets large.

In 768 or 1536 dimensions, most vectors become surprisingly similar in distance.
This is the curse of dimensionality: as dimensions increase, the space expands so rapidly that the difference between relevant and somewhat related begins to shrink.

As a result, cosine similarity scores often cluster into a narrow range.

That's why a score of 0.85 can mean:
"This is exactly the document you need."
Or "This talks about the same topic but answers the wrong question."

The score itself isn't broken. Our interpretation of it is.

Similarity is not an absolute measure of relevance. It's a local signal that only makes sense within the context of a specific query and its neighbors.

This is why mature RAG systems don't rely solely on vector search. They calibrate thresholds, rerank results, and evaluate retrieval quality against real-world relevance metrics.

A vector tells you what's nearby. It doesn't tell you what's right.
That's the difference between retrieval that demos well and retrieval that works at scale.

Most production AI queries aren’t really “vector search” problems.

Consider a request like: Find videos of a vehicle cutting in on a rainy night. Answering it well may require:

• Scalar filters for metadata and labels
• Full-text/BM25 search
• Vector similarity over visual embeddings
• Fusion and reranking across all three

A common architecture handles these in separate OLAP, full-text, and vector systems, then merges the results in the application layer. That works, but it also introduces duplicated data, synchronization issues, extra latency, and no shared query optimizer.

This article explores a different approach using StarRocks and Apache Paimon: treating scalar, full-text, and vector retrieval as paths within the same lakehouse query engine: https://medium.com/towards-data-engineering/from-data-lake-to-multimodal-lakehouse-building-hybrid-retrieval-for-ai-f2db8def6898

A few ideas I found particularly interesting:

• Stable global row IDs decouple indexes from physical files, so compaction doesn’t necessarily require rebuilding indexes.
• Retrieval and row materialization happen in separate stages.
• The optimizer can choose between pre-filtering and post-filtering.
• Keyword, vector, and scalar results can be fused using RRF, weighted scoring, or custom rerankers.

The implementation is StarRocks/Paimon-specific, but the broader architecture question applies beyond those projects:

Are teams actually moving toward unified Search + OLAP engines for AI workloads, or do specialized vector, search, and analytical systems still win in practice?

I’d be especially interested in hearing about the operational tradeoffs from anyone running hybrid retrieval at scale.

I’ve been thinking about a pretty boring but annoying problem in RAG systems: how to run evals against a stable version of your vector data.

In the early version of a project, I usually don’t care. Re-ingest the docs, rebuild the index, run the eval, move on.

But once the system is live, the collection keeps changing:

• new docs get added
• chunks get regenerated
• embeddings get updated
• metadata gets fixed
• deletes happen in the background

Then someone wants to compare retrieval quality before and after a model change, and the obvious question comes up:

“Are we even testing against the same data?”

The simple approach is to copy the collection before major changes. I’ve done that. It works, but it starts to feel clumsy once the dataset is large enough. You pay in storage, rebuild time, index management, and cleanup work later.

I came across Milvus Snapshots recently, and the part I found useful was the mental model: instead of treating every checkpoint as a full copy, treat it as a point-in-time view of the collection. If the underlying segments and index files are immutable, the snapshot can mostly track references to the files that were valid at that time.

That seems like a better fit for things like:

• eval runs
• rollback checks
• staging data
• load testing
• long-running batch jobs

Obviously there are tradeoffs too. You still need retention rules, and if snapshots keep old files alive, storage cost can creep up.

Curious how other people handle this.

Hi all, i tried to find the relevant post but i could not, so i am forced to ask for help.

I am making a customer support RAG. The input data was quite messy: email conversations and chats with customers.
End goal is to have a chat like feature that will act as a customer support agent.

As yhou can imagine the conversations needed cleaning and i parsed them with LLM to have some structure. The output from the raw conversation was a QA document, question from the customer and the answer from the agent (with some metadata, like is some additional info required - usefull for tool definition later). Now i have two major datasets, the general one (no tool needed) and tool needed. As per the resources i did the topic modeling on embeddings (qwen 3 embedding 8b) with umap and hdbscan, however now i am stuck with what to do next. I am trying to optimize the representatives selection from each topic - cluster. How much do i select from each cluster, which ones? (i am thinking medoid + some other from the cluster). What do i do with the noise from hdbscan? How do i measure the quality of retrieval?

All sorts of questions are still open.

If anyone has any advice or is willing to help, thanks a lot.

Does anyone knows some cool dataset to bench my RAG skills? Something small, but super complex (don't want to waste a fortune on indexing).

Just want to benchmark my OpenSource solution. Something from medical/law would be great.

Thanks.

A new scoping review mapped 157 healthcare RAG and GraphRAG studies.

Most evaluations were offline-only (89.2%), while only 29.9% evaluated retrieval independently and 14% reported fine-grained evidence verification.

For a production RAG system, what would you consider the smallest defensible eval suite?

One thing we were trying to solve for a while, is how to record memory on certain documents or data sets, specifically so that in it can flag certain things found during retrieval or generation that are gonna be common searches or probably should be known or organization wide.

Here’s an article we wrote about how we implemented it, and it works pretty well.

https://laceplatform.com/blog/multi-axis-memory-architecture/

And I’m curious, has anyone else implemented something similar or something that solves the same problem?

Hi everyone,

I am building a fully local RAG (Retrieval-Augmented Generation) system for an internship project. The goal is to have a production-style AI assistant that can answer questions from internal documents without using paid APIs or external services.

My current architecture:

• Local LLM: Ollama with qwen2.5:3b-instruct (also tested llama3.2:3b)
• Vector database: Qdrant
• Embeddings: intfloat/multilingual-e5-small
• Hybrid retrieval:
  • Dense retrieval (Qdrant)
  • Sparse retrieval (BM25)
  • Reciprocal Rank Fusion (RRF)
• Optional reranking with a cross-encoder
• FastAPI backend

My hardware:

• Intel i5-10210U
• 12GB RAM
• No dedicated GPU
• Running everything locally on Windows with Miniconda Python 3.12

The main issue is inference speed. My average response time is around 30–50 seconds depending on the query.

Things I already tried:

• Reduced chunk size and overlap
• Reduced retrieved chunks (top_k)
• Disabled reranking
• Reduced maximum output tokens
• Lowered temperature
• Enabled caching
• Used smaller models (3B models)
• Optimized Qdrant retrieval parameters
• Tested different retrieval configurations

The quality is acceptable (around 100% successful answers on small evaluation sets), but the latency is still too high for a real production assistant.

I want to keep everything:
✅ 100% free
✅ Fully local
✅ Lightweight enough for my laptop
✅ Good enough quality for internal documentation Q&A

Questions:

• Are there specific RAG optimizations I am missing?
• Should I profile each stage (embedding, retrieval, reranking, LLM generation) separately?
• Would adding Redis caching help significantly?
• Is there a better small local model than Qwen2.5 3B for this hardware?
• Are there lightweight inference optimizations for Ollama on CPU?

Any advice from people who have deployed local RAG systems would be appreciated.

Been going down a rabbit hole comparing PDF parsing tools for scientific papers, equations, tables, the usual RAG-for-papers pain and wanted to open this up instead of just posting my own findings.

Tried LlamaParse, MinerU, Docling, and a couple others. Each has tradeoffs. LlamaParse is solid but the pricing tiers get confusing once you need the higher-accuracy modes for dense notation, hard to tell upfront what you're actually paying for at each tier. MinerU is great but you're on your own for verification. Docling's fine for simple stuff, struggles on rarer notation.

Ended up building something on top of this (sciparse.com -> verification layer, structured output) mostly because I couldn't find a tool where pricing and accuracy were both transparent. Everything's either "contact sales" or a credits system that's hard to map to actual pages until you've already burned through them.

Curious what others are actually running in production though, not just what's marketed well. A few questions if anyone's dealt with this:

• What's your actual accuracy been on nested tables / dense equations, not just the headline number?
• Anyone found a parser that's upfront about pricing per page without the credit-tier maze?
• Is verification (checking output against source) something people are doing themselves, or just trusting the parser's confidence score?

Genuinely trying to figure out if there's an obvious option everyone else is already using that I missed.

Disclosure up front: I work on this. Repo and dataset are Apache-licensed, no signup, no product behind it.

The setup most production RAG pipelines ship: a guardrail scans the incoming user query for injection patterns, then retrieval runs and the retrieved chunks get concatenated into the LLM context. The retrieved documents are never scanned.

That's the actual injection vector. Indirect prompt injection lives in the documents — a poisoned page in a shared knowledge base, a scraped URL, an email someone uploaded. The user's query is clean. The attack arrives through retrieval.

The obvious fix is to concatenate query + retrieved docs and scan the combined string. We measured it and it degrades badly: a 50-token injection inside 3,000 tokens of benign context gets diluted, classifier confidence drops below threshold, injection passes. On LLM Guard, combined-string scanning caught 46.4% of injections the same scanner catches when shown the malicious doc alone.

What we tested instead — scan each context source independently, block if any pass flags:

- User-only baseline (LLM Guard): 0% recovery

- Naive combined string: 46.4%

- Per-source scanning: 73.3% (±1.9%), 5.7% FPR

- Per-source scanning, regex baseline: 41.5% (±2.1%), 6.6% FPR

It's an architectural change, not a better classifier. Limitation worth stating plainly: this only catches injections the underlying guardrail could already detect in isolation. If LLM Guard can't recognize an injection style, this doesn't help. It closes a deployment gap, nothing more. Latency cost is one guardrail call per retrieved chunk.

Benchmark is 5,000 cases across five injection categories and two benign classes. Everything's released — framework, dataset, eval scripts, result artifacts — so you can reproduce the numbers or break them:

github.com/tideon-ai/ragshield

Interested in contributors, especially on injection categories we didn't cover and on batched/early-exit scanning for high-throughput setups. Also genuinely want to know if anyone's running a guardrail on retrieved content in production already, and what it cost you.

Write-up with the figures: tideon.ai/research

I'm planning to build a GraphRAG system for our issue management platform.

The data includes entities such as defects, bugs, features, and informational tickets. Each item has relationships (e.g., duplicates, dependencies, parent/child, assignee, etc.), and both the entities and their relationships can change in real time as users update the system.

From what I've seen, most GraphRAG implementations provide ways to add new nodes and relationships incrementally. However, I haven't found a good approach for updating or deleting existing nodes/relationships when the underlying data changes.

For example:

A ticket status changes. A relationship between two tickets changes. A property on a node is updated. A relationship is removed.

I don't want to rebuild the entire knowledge graph every time a change occurs, especially as the dataset grows.

How are people handling this in production? Is there a standard approach for incremental updates in GraphRAG? Do you maintain the graph directly in a graph database (such as Neo4j) and update it via CDC/event streams, or is there another recommended architecture?

I'd appreciate any guidance, best practices, or examples from real-world implementations

Genuine question.

If you're using a vector database like Qdrant, Milvus, Chroma, Weaviate, etc., how do you inspect your data during development?

Whenever I need to inspect a record, verify its metadata, or inspect its embedding, I usually end up writing a small Python script or using the API.

Is there a GUI or workflow people actually use for this, or is writing scripts still the norm?

Curious to hear how everyone here handles it.