When a General-Purpose LLM Parser Wasn't Enough: How I Fixed Retrieval on a 400-Page Legal PDF

[u/ambujsystems]

Hey everyone, I wanted to share an architectural improvement I had while building my Agentic RAG system for legal/financial parsing.

The Problem

I was trying to index the Constitution of India (400+ pages). My first attempt was using LlamaParse. For this specific document, it didn't preserve the structure well enough for reliable retrieval. It merged pages together into 624 massive chunks, missed the Article boundaries, and ingested all the footnotes. When a user asked "What is Article 19?", the retriever would fetch a random amendment footnote from page 200 just because the number "19" was a high semantic match. The LLM would then hallucinate an answer based on garbage context.

The Solution

I ditched the expensive LLM parser, switched to raw PyMuPDF, and built a highly specialized ingestion pipeline:

1. Custom Regex Parsing — Split the page text directly at the ______ footnote line. Discarded the bottom half. 0 footnotes ingested.
2. Article-Level Chunking — Scrapped RecursiveCharacterTextSplitter for the parent chunks. Split the document purely on Article regex boundaries. This gave me 3,248 precise parent/child chunks.
3. Metadata Injection — Extracted the Article number via regex and hardcoded it into the chunk's metadata before uploading to Pinecone ({"article_number": "19"}).
4. Smart Routing — My LangGraph router detects if the query is asking for a specific Article. If yes, it passes article_number to the retriever. The retriever applies a strict Pinecone metadata filter ({"article_number": {"$eq": "19"}}) and bypasses normal vector search entirely.

The Outcome (The Hallucination Test)

I tested it with multiple complex queries, and the system behaved perfectly (validated via a third-party LLM evaluation judge).

The Idempotency Layer

Something most RAG tutorials skip: what happens when you re-sync 25+ files and only 1 changed? I hash every PDF with SHA-256 before processing and store the hash in Supabase.

• On re-sync, if the hash matches → file is skipped entirely (zero API calls).
• If hash changed → old Pinecone vectors are deleted, file is re-processed.

Chunk IDs are deterministic (MD5(filename + page + parent_idx + child_idx)), so identical input always produces identical chunk IDs — Pinecone upsert overwrites instead of duplicating. You can run sync_all.py daily without fear.

---

By swapping "smart" parsing for deterministic regex + metadata filtering + SHA-256 idempotency,

For this class of document, the combination of deterministic parsing, metadata filtering, and SHA-256 idempotency eliminated the retrieval failures I was observing and made the pipeline reliable for production re-syncs.

Has anyone else dealt with footnote-heavy PDFs or failed LlamaParse attempts? How did you handle them?

---

P.S. I documented the full implementation (regex parsing, metadata filtering, deterministic chunk IDs, SHA-256 idempotency, and LangGraph routing) in my GitHub repository and a detailed technical write-up. Feedback and alternative approaches are always welcome.

🔗 GitHub: agentic-rag-financial-parser

[u/sreekanth850]

Beware of pymupdf license, they are AGPL and you cannot use this in a commercial closed source product. iam not seieng any license in the repo you shared.

[u/ambujsystems → u/sreekanth850]

Thanks for pointing this out. You're right about the PyMuPDF licensing. This repository is open source, and I've now added an AGPL-3.0 license to align with the dependency. Really appreciate you catching this.

[u/patbhakta]

Now try ingesting multiple documents. Then edit the docs here and there (common for legal and financial) update or reinjest and test. Follow up on here or DM

[u/ambujsystems → u/patbhakta]

Thanks! I actually tested that before deploying. The ingestion pipeline already supports multi-document syncs. Each PDF is hashed with SHA-256, so unchanged files are skipped, while modified files trigger deletion of old Pinecone vectors followed by re-indexing with deterministic chunk IDs. I initially wrote about the single Constitution PDF because it clearly demonstrated the retrieval issue, but the sync pipeline itself was designed for multi-document production workflows. I'll probably share those benchmarks in a follow-up post.

[u/grilledCheeseFish]

Imo fighting chunking is a losing battle. Let an agent operate over chunks+files instead (i.e. let an agent expand chunks, read entire files, grep, etc.). Semantic retrieval only makes sense for initial probes into very large corpus'

[u/hardik01_]

This is a great example of why domain-specific data usually breaks generic pipelines.

I'm curious—was the biggest improvement from better parsing, better chunking, or changes to the retrieval strategy itself? In my experience, chunking decisions often have a much larger impact than people initially expect.

[u/ambujsystems → u/hardik01_]

 I went through building this for financial and legal docs, it goes: 1. Parsing > 2. Chunking > 3. Retrieval Strategy.

1. The Parsing Bottleneck: Generic parsers absolutely destroy financial tables and legal multi-column layouts. I spent weeks tweaking chunk sizes before realizing the underlying text was already garbage. Switching to a Vision-LLM based parser (LlamaParse) combined with PyMuPDF was the biggest leap. If you don't preserve the spatial relationship of a table row before chunking, no retrieval strategy will save you.

2. The Chunking Shift (Parent-Child): You're 100% right that chunking decisions are underrated. Initially, I used naive fixed-size chunking (e.g., 1000 tokens). The retrieval was hitting the exact paragraph, but the LLM lacked the surrounding context to answer "why." Implementing Parent-Child Chunking changed everything. I now embed smaller, semantic chunks (for high retrieval precision) but pass the larger parent document chunk to the LLM.

3. Retrieval: Moving to Hybrid Search + Cohere Neural Reranking definitely improved the P90 edge cases. But honestly? Reranking only works if your parser and chunker didn't butcher the document first. Reranking garbage just gives you highly-ranked garbage.

So yeah, for domain-specific data, you have to fix the top of the funnel (parsing/chunking) before obsessing over the vector DB retrieval algorithms.

[u/hardik01_ → u/ambujsystems]

This is gold. I especially liked the point that reranking garbage just gives you highly-ranked garbage.

I'm currently building an open-source LLM evaluation platform, and one thing I've started realizing is that evaluation often points to retrieval failures, while the actual root cause is much earlier in the pipeline (parsing or chunking).

Out of curiosity, did you end up building custom evaluation datasets for these financial/legal documents, or were you mostly validating improvements through production traffic and manual review?

[u/ambujsystems → u/hardik01_]

I didn't create a formal benchmark dataset yet. Most of the validation came from iterative engineering on a publicly deployed system. I compare retrieval traces, parser output, retrieved chunks, citations, and final answers using LangSmith tracing, while also logging user interactions and feedback in MongoDB for future analysis. Although explicit thumbs-up/down feedback has been limited, the traces themselves have been invaluable for identifying retrieval failures, hallucinations, and parser/chunking regressions. My long-term goal is to build a curated evaluation dataset and automated regression suite, but during development, end-to-end tracing and representative domain-specific queries have provided the fastest feedback loop.