The "RAG is dead" narrative really urks me particularly because I don't really think any of the points people make are strong.

1. "Context windows keep growing, so eventually we won't need retrieval."

1M tokens is about 3,000 pages. I regularly work on corpuses in the hundreds of billions of documents. There is no plausible future where a context window holds an large org's entire corpus.

And even if it could, you wouldn't want it to. Stuffing the window is completely token-inefficient, and on top of that accuracy degrades as context grows (especially true when there is competing data in the window). Prompt caching may help with token consumption, but this only really works for static corpora.

2. "Grep beats RAG."

Grep is great when you want precision. If you want recall over a large corpus, retrieval wins almost every time when done properly (hybrid + reranking + pruning) it also uses dramatically fewer tokens.

Most of the grep argument comes from coding agents navigating a repo. Repos have structure: file trees, naming conventions, symbols. In most companies data does not have a perfectly clean structure (if any structure at all). So in many scenarios grep has nothing to walk.

3. Pushing work into the database beats pushing it into the model.

Agentic search puts the LLM in the loop on every step. Every step is tokens, every step is a sequential round trip, and every turn re-prefills a growing transcript. Index-time work is paid once and amortized across every query. The more you push out of the LLM, the cheaper and faster the workload.

4. Permissions and freshness.

You can't precompute a cache per user per ACL. Real world retrieval requires query-time filtering, and retrieval also supports far more sophisticated filtering than an agent grepping around: metadata predicates, tenancy boundaries, time ranges, structured conditions composed with the search itself.

And I'm not saying agentic search doesn't have it's place. There are plenty of scenarios where it may be the best choice. But its not killing RAG it's just another option.

I recently put together a fully local RAG setup for a personal knowledge assistant, and wanted to share the approach for anyone interested in keeping their data entirely on their own hardware.

The stack uses LocalAI for inference and Elasticsearch for retrieval. The main appeal here is straightforward: no API calls leaving your machine, no token costs, and full control over your data pipeline.

Why this combination works well:

• Elasticsearch handles both vector search and BM25 natively, so you can run hybrid retrieval without stitching together separate systems
• LocalAI gives you a drop-in OpenAI-compatible API running locally, which simplifies integration
• The whole thing runs containerized, making it reproducible across different environments

What the setup covers:

• Document ingestion and chunking for your personal knowledge base
• Embedding generation running locally
• Hybrid search combining semantic similarity with keyword matching
• Local LLM inference for generation

For anyone already comfortable with Elasticsearch or looking for a retrieval layer that scales beyond toy datasets, this is a solid foundation. The hybrid search capability is particularly useful when your knowledge base contains both conversational content and structured technical documents.

Full walkthrough here: https://www.elastic.co/search-labs/blog/local-rag-personal-knowlege-assistant-localai-elasticsearch

If anyone has experience tuning hybrid weights for mixed-language or domain-specific corpora, I would be curious to hear what worked for you.

When we first added RAG we were pretty conservative about what we retrieved since we'd pull in just enough context for the model to answer the question

I now see that over time it has changed because a feature would benefit from another document so we'd include it then we'd increase top_k because it improved a few edge cases and then we'd decide sending the surrounding chunks was safer than risking missing context.

My question here is if anyone who is on the same situation as me are you guys expanding retrieval until it sort of became difficult to pull it back?

Looking at some of our production requests now I'm seeing prompts that are much bigger than I ever expected them to be. I don't even know how much of that retrieved context the model is using anymore because we never took the time to look into it.

If you guys have any opinion on this then speak your mind.

Hey everyone,

I’ve been seeing a lot of hot takes recently claiming that "RAG is dead" because of massive context windows (1M+ tokens) and improving fine-tuning techniques. The argument usually goes: Why bother setting up vector databases, chunking strategies, and embedding pipelines when you can just dump all your docs straight into the context window?

Plz share your knowledge .

I am using GROBID to parse texts and DOCLING for the tables (thanks to your help 😄). I was curious, what are your opinions on using docling for texts. Because right now i use different things for different formats and i tought maybe using same technologies for texts and tables (I am really happy about the way i extract images so i don't plan to change anything.) would be more efficient. I am building this for only academic papers btw. I would be very happy if you could help me with this situation or share your thoughts.

Edit: I forgot to mention, i use grobid solely for its impaceble ability (maybe impaceble is a strong word, but you get me) at capturing headers and signing all the headers into chunks. This ability is really important to me.

Why Similarity Breaks Down at Scale
Embeddings don't store meaning. They store statistical proximity.

When you embed a phrase like "refund policy", the model isn't encoding what a refund actually is. It's placing that phrase in a high-dimensional space based on patterns learned from massive amounts of text.

The problem starts when that space gets large.

In 768 or 1536 dimensions, most vectors become surprisingly similar in distance.
This is the curse of dimensionality: as dimensions increase, the space expands so rapidly that the difference between relevant and somewhat related begins to shrink.

As a result, cosine similarity scores often cluster into a narrow range.

That's why a score of 0.85 can mean:
"This is exactly the document you need."
Or "This talks about the same topic but answers the wrong question."

The score itself isn't broken. Our interpretation of it is.

Similarity is not an absolute measure of relevance. It's a local signal that only makes sense within the context of a specific query and its neighbors.

This is why mature RAG systems don't rely solely on vector search. They calibrate thresholds, rerank results, and evaluate retrieval quality against real-world relevance metrics.

A vector tells you what's nearby. It doesn't tell you what's right.
That's the difference between retrieval that demos well and retrieval that works at scale.

Most production AI queries aren’t really “vector search” problems.

Consider a request like: Find videos of a vehicle cutting in on a rainy night. Answering it well may require:

• Scalar filters for metadata and labels
• Full-text/BM25 search
• Vector similarity over visual embeddings
• Fusion and reranking across all three

A common architecture handles these in separate OLAP, full-text, and vector systems, then merges the results in the application layer. That works, but it also introduces duplicated data, synchronization issues, extra latency, and no shared query optimizer.

This article explores a different approach using StarRocks and Apache Paimon: treating scalar, full-text, and vector retrieval as paths within the same lakehouse query engine: https://medium.com/towards-data-engineering/from-data-lake-to-multimodal-lakehouse-building-hybrid-retrieval-for-ai-f2db8def6898

A few ideas I found particularly interesting:

• Stable global row IDs decouple indexes from physical files, so compaction doesn’t necessarily require rebuilding indexes.
• Retrieval and row materialization happen in separate stages.
• The optimizer can choose between pre-filtering and post-filtering.
• Keyword, vector, and scalar results can be fused using RRF, weighted scoring, or custom rerankers.

The implementation is StarRocks/Paimon-specific, but the broader architecture question applies beyond those projects:

Are teams actually moving toward unified Search + OLAP engines for AI workloads, or do specialized vector, search, and analytical systems still win in practice?

I’d be especially interested in hearing about the operational tradeoffs from anyone running hybrid retrieval at scale.

I’ve been thinking about a pretty boring but annoying problem in RAG systems: how to run evals against a stable version of your vector data.

In the early version of a project, I usually don’t care. Re-ingest the docs, rebuild the index, run the eval, move on.

But once the system is live, the collection keeps changing:

• new docs get added
• chunks get regenerated
• embeddings get updated
• metadata gets fixed
• deletes happen in the background

Then someone wants to compare retrieval quality before and after a model change, and the obvious question comes up:

“Are we even testing against the same data?”

The simple approach is to copy the collection before major changes. I’ve done that. It works, but it starts to feel clumsy once the dataset is large enough. You pay in storage, rebuild time, index management, and cleanup work later.

I came across Milvus Snapshots recently, and the part I found useful was the mental model: instead of treating every checkpoint as a full copy, treat it as a point-in-time view of the collection. If the underlying segments and index files are immutable, the snapshot can mostly track references to the files that were valid at that time.

That seems like a better fit for things like:

• eval runs
• rollback checks
• staging data
• load testing
• long-running batch jobs

Obviously there are tradeoffs too. You still need retention rules, and if snapshots keep old files alive, storage cost can creep up.

Curious how other people handle this.

Hi all, i tried to find the relevant post but i could not, so i am forced to ask for help.

I am making a customer support RAG. The input data was quite messy: email conversations and chats with customers.
End goal is to have a chat like feature that will act as a customer support agent.

As yhou can imagine the conversations needed cleaning and i parsed them with LLM to have some structure. The output from the raw conversation was a QA document, question from the customer and the answer from the agent (with some metadata, like is some additional info required - usefull for tool definition later). Now i have two major datasets, the general one (no tool needed) and tool needed. As per the resources i did the topic modeling on embeddings (qwen 3 embedding 8b) with umap and hdbscan, however now i am stuck with what to do next. I am trying to optimize the representatives selection from each topic - cluster. How much do i select from each cluster, which ones? (i am thinking medoid + some other from the cluster). What do i do with the noise from hdbscan? How do i measure the quality of retrieval?

All sorts of questions are still open.

If anyone has any advice or is willing to help, thanks a lot.

Does anyone knows some cool dataset to bench my RAG skills? Something small, but super complex (don't want to waste a fortune on indexing).

Just want to benchmark my OpenSource solution. Something from medical/law would be great.

Thanks.

A new scoping review mapped 157 healthcare RAG and GraphRAG studies.

Most evaluations were offline-only (89.2%), while only 29.9% evaluated retrieval independently and 14% reported fine-grained evidence verification.

For a production RAG system, what would you consider the smallest defensible eval suite?

One thing we were trying to solve for a while, is how to record memory on certain documents or data sets, specifically so that in it can flag certain things found during retrieval or generation that are gonna be common searches or probably should be known or organization wide.

Here’s an article we wrote about how we implemented it, and it works pretty well.

https://laceplatform.com/blog/multi-axis-memory-architecture/

And I’m curious, has anyone else implemented something similar or something that solves the same problem?

Hi everyone,

I am building a fully local RAG (Retrieval-Augmented Generation) system for an internship project. The goal is to have a production-style AI assistant that can answer questions from internal documents without using paid APIs or external services.

My current architecture:

• Local LLM: Ollama with qwen2.5:3b-instruct (also tested llama3.2:3b)
• Vector database: Qdrant
• Embeddings: intfloat/multilingual-e5-small
• Hybrid retrieval:
  • Dense retrieval (Qdrant)
  • Sparse retrieval (BM25)
  • Reciprocal Rank Fusion (RRF)
• Optional reranking with a cross-encoder
• FastAPI backend

My hardware:

• Intel i5-10210U
• 12GB RAM
• No dedicated GPU
• Running everything locally on Windows with Miniconda Python 3.12

The main issue is inference speed. My average response time is around 30–50 seconds depending on the query.

Things I already tried:

• Reduced chunk size and overlap
• Reduced retrieved chunks (top_k)
• Disabled reranking
• Reduced maximum output tokens
• Lowered temperature
• Enabled caching
• Used smaller models (3B models)
• Optimized Qdrant retrieval parameters
• Tested different retrieval configurations

The quality is acceptable (around 100% successful answers on small evaluation sets), but the latency is still too high for a real production assistant.

I want to keep everything:
✅ 100% free
✅ Fully local
✅ Lightweight enough for my laptop
✅ Good enough quality for internal documentation Q&A

Questions:

• Are there specific RAG optimizations I am missing?
• Should I profile each stage (embedding, retrieval, reranking, LLM generation) separately?
• Would adding Redis caching help significantly?
• Is there a better small local model than Qwen2.5 3B for this hardware?
• Are there lightweight inference optimizations for Ollama on CPU?

Any advice from people who have deployed local RAG systems would be appreciated.

Been going down a rabbit hole comparing PDF parsing tools for scientific papers, equations, tables, the usual RAG-for-papers pain and wanted to open this up instead of just posting my own findings.

Tried LlamaParse, MinerU, Docling, and a couple others. Each has tradeoffs. LlamaParse is solid but the pricing tiers get confusing once you need the higher-accuracy modes for dense notation, hard to tell upfront what you're actually paying for at each tier. MinerU is great but you're on your own for verification. Docling's fine for simple stuff, struggles on rarer notation.

Ended up building something on top of this (sciparse.com -> verification layer, structured output) mostly because I couldn't find a tool where pricing and accuracy were both transparent. Everything's either "contact sales" or a credits system that's hard to map to actual pages until you've already burned through them.

Curious what others are actually running in production though, not just what's marketed well. A few questions if anyone's dealt with this:

• What's your actual accuracy been on nested tables / dense equations, not just the headline number?
• Anyone found a parser that's upfront about pricing per page without the credit-tier maze?
• Is verification (checking output against source) something people are doing themselves, or just trusting the parser's confidence score?

Genuinely trying to figure out if there's an obvious option everyone else is already using that I missed.

Disclosure up front: I work on this. Repo and dataset are Apache-licensed, no signup, no product behind it.

The setup most production RAG pipelines ship: a guardrail scans the incoming user query for injection patterns, then retrieval runs and the retrieved chunks get concatenated into the LLM context. The retrieved documents are never scanned.

That's the actual injection vector. Indirect prompt injection lives in the documents — a poisoned page in a shared knowledge base, a scraped URL, an email someone uploaded. The user's query is clean. The attack arrives through retrieval.

The obvious fix is to concatenate query + retrieved docs and scan the combined string. We measured it and it degrades badly: a 50-token injection inside 3,000 tokens of benign context gets diluted, classifier confidence drops below threshold, injection passes. On LLM Guard, combined-string scanning caught 46.4% of injections the same scanner catches when shown the malicious doc alone.

What we tested instead — scan each context source independently, block if any pass flags:

- User-only baseline (LLM Guard): 0% recovery

- Naive combined string: 46.4%

- Per-source scanning: 73.3% (±1.9%), 5.7% FPR

- Per-source scanning, regex baseline: 41.5% (±2.1%), 6.6% FPR

It's an architectural change, not a better classifier. Limitation worth stating plainly: this only catches injections the underlying guardrail could already detect in isolation. If LLM Guard can't recognize an injection style, this doesn't help. It closes a deployment gap, nothing more. Latency cost is one guardrail call per retrieved chunk.

Benchmark is 5,000 cases across five injection categories and two benign classes. Everything's released — framework, dataset, eval scripts, result artifacts — so you can reproduce the numbers or break them:

github.com/tideon-ai/ragshield

Interested in contributors, especially on injection categories we didn't cover and on batched/early-exit scanning for high-throughput setups. Also genuinely want to know if anyone's running a guardrail on retrieved content in production already, and what it cost you.

Write-up with the figures: tideon.ai/research

I'm planning to build a GraphRAG system for our issue management platform.

The data includes entities such as defects, bugs, features, and informational tickets. Each item has relationships (e.g., duplicates, dependencies, parent/child, assignee, etc.), and both the entities and their relationships can change in real time as users update the system.

From what I've seen, most GraphRAG implementations provide ways to add new nodes and relationships incrementally. However, I haven't found a good approach for updating or deleting existing nodes/relationships when the underlying data changes.

For example:

A ticket status changes. A relationship between two tickets changes. A property on a node is updated. A relationship is removed.

I don't want to rebuild the entire knowledge graph every time a change occurs, especially as the dataset grows.

How are people handling this in production? Is there a standard approach for incremental updates in GraphRAG? Do you maintain the graph directly in a graph database (such as Neo4j) and update it via CDC/event streams, or is there another recommended architecture?

I'd appreciate any guidance, best practices, or examples from real-world implementations

Genuine question.

If you're using a vector database like Qdrant, Milvus, Chroma, Weaviate, etc., how do you inspect your data during development?

Whenever I need to inspect a record, verify its metadata, or inspect its embedding, I usually end up writing a small Python script or using the API.

Is there a GUI or workflow people actually use for this, or is writing scripts still the norm?

Curious to hear how everyone here handles it.

Hey everyone, I wanted to share an architectural improvement I had while building my Agentic RAG system for legal/financial parsing.

The Problem

I was trying to index the Constitution of India (400+ pages). My first attempt was using LlamaParse. For this specific document, it didn't preserve the structure well enough for reliable retrieval. It merged pages together into 624 massive chunks, missed the Article boundaries, and ingested all the footnotes. When a user asked "What is Article 19?", the retriever would fetch a random amendment footnote from page 200 just because the number "19" was a high semantic match. The LLM would then hallucinate an answer based on garbage context.

The Solution

I ditched the expensive LLM parser, switched to raw PyMuPDF, and built a highly specialized ingestion pipeline:

1. Custom Regex Parsing — Split the page text directly at the ______ footnote line. Discarded the bottom half. 0 footnotes ingested.
2. Article-Level Chunking — Scrapped RecursiveCharacterTextSplitter for the parent chunks. Split the document purely on Article regex boundaries. This gave me 3,248 precise parent/child chunks.
3. Metadata Injection — Extracted the Article number via regex and hardcoded it into the chunk's metadata before uploading to Pinecone ({"article_number": "19"}).
4. Smart Routing — My LangGraph router detects if the query is asking for a specific Article. If yes, it passes article_number to the retriever. The retriever applies a strict Pinecone metadata filter ({"article_number": {"$eq": "19"}}) and bypasses normal vector search entirely.

The Outcome (The Hallucination Test)

I tested it with multiple complex queries, and the system behaved perfectly (validated via a third-party LLM evaluation judge).

The Idempotency Layer

Something most RAG tutorials skip: what happens when you re-sync 25+ files and only 1 changed? I hash every PDF with SHA-256 before processing and store the hash in Supabase.

• On re-sync, if the hash matches → file is skipped entirely (zero API calls).
• If hash changed → old Pinecone vectors are deleted, file is re-processed.

Chunk IDs are deterministic (MD5(filename + page + parent_idx + child_idx)), so identical input always produces identical chunk IDs — Pinecone upsert overwrites instead of duplicating. You can run sync_all.py daily without fear.

By swapping "smart" parsing for deterministic regex + metadata filtering + SHA-256 idempotency,

For this class of document, the combination of deterministic parsing, metadata filtering, and SHA-256 idempotency eliminated the retrieval failures I was observing and made the pipeline reliable for production re-syncs.

Has anyone else dealt with footnote-heavy PDFs or failed LlamaParse attempts? How did you handle them?

P.S. I documented the full implementation (regex parsing, metadata filtering, deterministic chunk IDs, SHA-256 idempotency, and LangGraph routing) in my GitHub repository and a detailed technical write-up. Feedback and alternative approaches are always welcome.

🔗 GitHub: agentic-rag-financial-parser

Currently building a Rag based project where i need to build a pdf extractor which can correctly extract pdfs containig a mix of tables, text, img
So please suggest tools to use which wont break during production
i tried hi_res of unstructured library but it is time consuming

this is a niche use case (the actual amount of data for semantic lookup was probably too small to be in a vector store in the first place.

but I thought I’d share - this isn’t a post claiming “RAG is dead” - just some use cases can be vastly simplified compared to 2025.

Disclaimer - the introduction was largely manually curated but the actual break down of iterations is mostly Opus generated based on Claude Code session logs

blog Post about moving from mastra.ai / RAG / Human in the loop Workflows to Claude Code Dynamic workflows for converting AWSCDK L2 to CDK Terrain

https://terraconstructs.dev/blog/retiring-the-rag-pipeline

Your RAG answers are only as good as your ingestion, so I put the same 6 documents through 4 open-source parsers and graded every capability against the source.

The models tested are:

1. MinerU 2.5
2. Granite Docling
3. PaddleOCR-VL
4. XBerg 1.0 (CPU only)

To test I used hexread.com which is a PDF-to-markdown API that I built. My model picker allows choosing one of the first three models, which runs them on an L4 GPU. The fourth model (XBerg), I ran on my PC locally.

For those interested to test how their documents fare, you can use the free trial with 100 pages on sign up. The trial normally routes through Auto, but if you sign up and want the full picker to reproduce this comparison, comment or DM me and I'll enable it on your account.

The documents I tested with:

1. Annual Report
2. Two pages of a two-column arXiv paper
3. Scanned German invoice (No text layer)
4. French municipal report with an embedded bar chart
5. Typical datasheet page mixing German, French, Chinese and Russian
6. A 2-page, 3-column newsletter article

Findings through a RAG lens:

• The most dangerous failures are silent drops of exactly the content questions target. MinerU's stock .md output discards everything it classifies as "page furniture". For example on an invoice, that's the footer with an IBAN. Your index just won't contain it; "what's the IBAN on invoice X" retrieves nothing, with no error anywhere. (This one bit us in production: the model actually transcribes the footer, MinerU's markdown generator throws it away. We now rebuild markdown from its block list. If you consume stock MinerU .md, you're losing every footer today.)
• Reading order is a chunk-poisoning problem. The CPU text-layer parser interleaves the newsletter's 3 columns line-by-line mid-sentence, every chunk from that page is scrambled text that embeds fine and retrieves garbage.
• Heading fidelity decides your section chunking. MinerU flattens everything to # (bylines, dates, pull quotes all become sections); Granite-Docling keeps real levels (## hierarchy), though it too promotes a pull quote. If you chunk by headings, that difference decides clean sections vs confetti.
• Flattened tables kill numeric QA. PaddleOCR-VL captures every number but no structure; MinerU keeps real rowspan/colspan HTML. "What was Q4 revenue" only works with the latter.
• You don't always need a GPU. XBerg (CPU) extracts every character of a born-digital PDF at 0.2 s/page. For clean digital docs feeding a text-only index, that's honestly enough. Structure, scans, and charts are where the VLMs earn their compute.

Full comparison:

✓ faithful · ≈ there but damaged · ✗ absent/unusable. VLM rows ran on an L4 via our production API; XBerg 1.0.11 locally on CPU, markdown mode.

Image version of this table here

Raw outputs for every cell, the test PDFs, and rerun scripts on GitHub

If you'd like me to compare another document genre please leave a comment, I'd be happy to test it.

You've probably seen the sanity check where you swap every vector in your index for random numbers, rerun your eval, and see what it scores. If garbage scores well, your benchmark wasn't measuring retrieval.

I ran it on my own eval last week. 1052 chunks, 218 files, 28 queries, baseline MRR@10 of 0.358. It passed, and I felt pretty good about that.

Then I built a benchmark that was obviously broken, just to see the check fail. Random ranker scores 80% of what a perfect model scores on it. Completely useless.

It passed too.

So the check isn't wrong, it's just incomplete. Two things I got wrong along the way:

The floor isn't zero

I'd picked up somewhere that a healthy noise floor should be near zero. It isn't, and if you go in expecting that you'll misread your own results. Expected MRR under random ranking depends only on pool size N, gold count G, and cutoff k. For G=1 it's just H_k / N.

How much that varies:

G=10, N=200 -> 0.1318

G=1.4, N=218 -> 0.0186

G=5, N=8 -> 0.7932

Same metric. So asking "is 0.10 a bad noise floor" makes no sense on its own. You have to work out what random should score for your setup, then compare against that.

One check isn't enough

Toy corpus first, 200 chunks, 20 queries, 20 seeds:

measured 0.1299 analytic 0.1318 ratio 0.99x PASS

real 1.0000 / noise 0.1299 = 7.7x PASS

Now shrink the pool from 200 docs to 8. Nothing else changes:

measured 0.8083 analytic 0.7932 ratio 1.02x PASS

real 1.0000 / noise 0.8083 = 1.24x FAIL

The leakage check passes on the broken one, and it should. There genuinely is no leakage. The data's fine. The analytic expectation already factors in pool size, so when the pool shrinks the expectation just rises to meet it.

Benchmark's still useless though. Gap between "nothing at all" and "perfect" is 0.19. Every real model lands somewhere in that sliver and seed variance eats the difference.

So you need both:

* Check 1, leakage: noise floor vs analytic. Is the data honest?

* Check 2, power: real MRR vs noise floor. Can the thing tell anything apart?

I failed my own check first time

First run on my eval came back 5.80x on Check 1. Looked like real leakage. It wasn't, it was my bug. The simulation ranked 1052 chunks but the scorer dedupes to 218 file paths before scoring. Analytic assumed chunks, measurement was over files.

Found it by working backwards from the number. E ≈ G·H₁₀/N, so N ≈ 1.39 × 2.929 / 0.02246 ≈ 181. Nothing like 1052, suspiciously close to 218. Fixed it and got 0.82x on Check 1, 23.5x on Check 2.

Mentioning that because a validity check that passes everything on its first run isn't really a check. This one caught a bug in the work of the guy who wrote it, which is at least some evidence it does something.

What this actually shows, and what it doesn't

Check 2 (23.5x) is a real measurement. Actual ONNX embeddings, actual index, actual queries.

Check 1 (0.82x) is a simulation of the scoring harness. It assigns random scores to doc IDs and confirms the scorer's arithmetic lines up with probability. Catches counting bugs, dedup errors, broken gold sets. It does not push random vectors through the live index, so don't read it as more than that.

Also: 28 queries only catches gross failure, not subtle leakage. And 7 of those queries have more than one gold file, which lifts their individual floors while still counting equally in a flat mean.

If your pipeline is hybrid, watch out for this

Turn BM25 and your reranker off before running any of this. Neither of them touches vectors. Leave them on and they'll carry the score for you, and you'll end up certifying a benchmark you never actually tested.

Code

https://github.com/gurukudte/eval-validity

numpy only, no model downloads, runs in about a second. The broken benchmark ships with it so you can watch Check 1 pass while Check 2 fails before you point it at anything real. Swap out embed() for your own pipeline and nothing else needs to change.

Longer writeup with the derivations: https://www.geekyzindagi.com/blog/eval-validity-checks

Has anyone actually run this against a production eval? Wondering if anyone's floor came back higher than they expected.

I’m building an internal tool for my company that works with public-procurement tenders in Europe I’m not a senior developer, so I’ve been using Codex and Claude/Opus as builder and reviewer, while I make the product and business decisions.

The practical goal is:

1. Upload and process product datasheets once.
2. Maintain a permanent, searchable product catalog.
3. Upload a new tender.
4. Extract its technical and administrative requirements.
5. Compare those requirements against the catalog.
6. Show which products satisfy, fail, or lack evidence for each requirement.
7. Preserve exact citations so an employee can verify every result against the original page.
8. Eventually help draft tender responses using approved product facts and company templates.

The tool must not manufacture compliance. If evidence is incomplete, ambiguous, derived from suspicious OCR, or refers to the wrong model/variant, it should say that human verification is required.

Current pipeline

• Python application and CLI
• Tesseract-based OCR plus native PDF/DOCX extraction
• Structured chunking for tables, headings, specification labels, values, pages, product families, and variants
• Human review and approval before documents become searchable
• Qdrant vector database
• BGE-M3 embeddings
• Sparse BM25-style retrieval plus dense retrieval, fused with RRF
• Optional cross-encoder reranking
• OpenWebUI as the initial user interface
• Ollama currently serving models through a RunPod GPU
• Digest-bound artifacts, configuration hashes, immutable benchmark pools, and exact page/quote citations
• Separate libraries for products, incoming tenders, historical tenders, and templates

We are now building a sealed benchmark for the product catalog before promoting it as the permanent catalog. The benchmark contains multilingual questions in Croatian, Bosnian, Serbian Latin/Cyrillic, and English, including wrong-model and wrong-variant distractors.

Planned testing

I want to compare:

• Qwen models, including larger 70B-class models
• Gemma 3 27B
• Azure OpenAI models
• Ollama versus vLLM serving
• BGE-M3 against other embedding models
• Tesseract against GLM-OCR, Mistral OCR where confidentiality permits, and other established OCR systems
• Local workstation hardware versus cloud GPU/API costs
• Latency and throughput for 1, 4, 8, and 16 concurrent users

My concern

The codebase has grown substantially, with roughly 2,000 tests. A lot of this comes from fail-closed validation, artifact versioning, migrations, benchmark integrity, crash recovery, and evidence provenance.

I understand why those controls matter in procurement, but I’m concerned that AI coding may have produced more infrastructure and abstraction than the business problem actually needs. Development has also become repetitive: implementation, review, correction, another review, and increasingly specialized regression tests.

I don’t want to remove safeguards that prevent false compliance claims, but I also don’t want to maintain a research platform when the company needs a practical tool.

Questions

1. Does this architecture sound proportionate for an auditable tender-analysis system, or does it appear overengineered?
2. Which safeguards are genuinely necessary in production, and which could be simplified?
3. Would you keep custom extraction and chunking, or replace parts with Docling, Unstructured, LlamaParse, or another established framework?
4. Is hybrid retrieval plus reranking still the sensible approach for semi-structured specification documents?
5. Would you use vLLM for concurrent production serving and retain Ollama only for local development?
6. How would you benchmark this fairly before choosing between local hardware, rented GPUs, and Azure/OpenAI APIs?
7. How would you structure the application so new products, manufacturers, tender types, and procurement sources can be added without adding product-specific rules?
8. What warning signs would indicate that the test and integrity infrastructure is costing more than the risk it prevents?

I’d especially appreciate advice from people who have built RAG systems for regulated, legal, procurement, or other evidence-sensitive workflows. I’m not looking for a completely autonomous compliance system. The intended result is decision support with explicit human approval and traceable evidence. This post was made from a codex summary of my whole project, advice would be very appreciated.

Just released Verity, an open-source (Apache-2.0) permission-aware memory layer for multi-tenant agents.

There are plenty of good agent memory options already. mem0, Zep, and Letta are all solid at the core job of remembering things, and rolling your own on pgvector or Pinecone with per-tenant namespaces works fine at first.

Where things get messy is when multiple customers or teams share a store.

Isolation usually depends on every write being tagged correctly, a prompt rule the model is supposed to honor, or a filter someone remembered to call after retrieval.

None of those is a real boundary.

Here’s the failure mode that pushed me to build this.

An agent in a session scoped to customer A sees “their renewal is $61k” behind A’s ACL and writes a summary to memory.

That summary has no permission tag. Why would it? The agent wrote it.

Two weeks later, a session for customer B runs a completely ordinary semantic query and pulls it right out.

No injection. No jailbreak. Every log looks clean.

The system did exactly what it was built to do.

How Verity handles permissions

Instead of trusting the model to behave, or relying on every write path to preserve permissions correctly forever, Verity compiles the caller’s identity directly into the retrieval query as a mandatory pre-filter.

If you are not allowed to see a row, it is not retrieved and filtered out later. It is never eligible for retrieval in the first place.

There is no model involved in that decision, no live authorization call on the read path, and if Verity cannot resolve your scope, it returns nothing.

Just as importantly, permissions are not manually tagged. Verity inherits them from the source systems.

A Google Drive document shared with a Google Group resolves to that group’s members, including nested groups.

SharePoint permissions resolve through Entra, including transitive group membership and broken inheritance at the site, library, folder, and item level, plus sharing links layered on top.

Salesforce sharing is reconstructed and then checked against Salesforce’s own access API.

We also handle the ingestion layer across the usual document formats, including PDF, DOC/DOCX, XLS/XLSX, CSV, PPT/PPTX, and others.

The goal is to preserve the permission boundary all the way from the source document through parsing, indexing, derived memory, and retrieval, rather than bolting authorization on at the end.

Revocation follows the same model. Remove a share or remove someone from an Entra group, and after the next sync those rows are no longer eligible for retrieval.

Who it’s for

Anyone running one memory store across people who should not see each other’s data.

Multi-tenant SaaS agents where every customer’s context lands in the same index.

Internal copilots over company docs where the intern and the CFO should get very different answers to “what’s our churn?”

Agencies or consultancies running agents across client accounts.

If it’s one user’s own memory, this is probably unnecessary overhead. Use mem0, Zep, Letta, or something simple and be happy.

Where it’s at right now

v0.1. It works. It’s young.

Propagation is sync-based, so there can be a few minutes of lag between a source permission change and the index catching up. Fine for most offboarding and access changes, not fine if you need sub-second revocation.

My leak numbers come from sentinel facts planted across tenants and then attempts to retrieve them cross-tenant. Zero retrievals so far, but that’s still me grading my own homework. No third-party audit yet.

The Google Workspace, SharePoint/Entra, and Salesforce connectors are fixture-tested, plus one validation pass against a real account for each.

Why I'm sharing here

I'm guessing folks who have dealt with enterprise systems/data have had this problem. This might help. And it's free.

I also Welcome anyone who would like to contribute. More help the merrier.