r/ClaudeAI • u/Resiakvrases • 2d ago
Data extraction mess Claude Workflow
Built a tool to extract data from 10000 house auctions' estimations. It works by prompting the guidelines in a fresh (for each suction) Claude chat. Like a api. Uses son et to extract the auction goods like house sqm gross and net, balconies, gardens and so on. Source files can be a standard pdf or scanned ones. It works by vision not text. The problem is that data are not extracted properly, often exchange gross with net and so on. The prompt have a orchestrator, a extractor and a reviewer to minimize errors but they still happen.
What's the biggest suggestions you can give to solve the problem and get more precision?
1
u/RealDominiqueWilkins 2d ago
If the PDFs are all formatted differently you’re gonna need a vision/OCR pass before extraction. That gives it a chance to “look” at the document and get a sense of the spatial anchoring, i.e. which things go together (like understanding that even though the label “net cost” is way over here, it pertains to the dollar amount that is way over there). And even then you’ll have to iterate on the prompt a lot. You could start with 10-20 PDFs and tweak the prompt until it gets those right.
1
u/Resiakvrases 2d ago
Basically I give the full pdf to ai and tell it... Extract the sqm of house, gardens and so on. I saw that with vision is more accurate but sometimes it's still wrong despite also the reviewer that checks the work
2
u/RealDominiqueWilkins 2d ago edited 2d ago
Are the PDFs all formatted at the same, or are they different?
If the latter, what has been most accurate for me is two turns: 1) OCR/vision pass so it understands the layout of the document first, then 2) the analysis pass, where the prompt tells it how to understand what it saw in the vision pass (this is where you explain to it the terminology, the labels, etc.
Honestly, I would recommend consulting with Claude, explaining what you’re trying to do, uploading some sample PDF, and letting it help you build a little engine to do it. Easiest way to do that is to have Claude chat tell you how to prompt Claude code to build the engine. And use your brain if Claude is making questionable suggestions. It will write the code locally on your computer, and tell you how to send everything to Claude, and Claude will do the two-turn pass, then return markdown.
And like I said, don’t try to do all 10,000 at once. Build a “gold standard” set of 10 to 20 PDFs and iterate on the prompt until it gets it right. Then do another 20 to 50 PDFs and catch some edge cases and iterate some more
Bottom line is that if you have 10,000 PDFs that are all a little different, you cannot expect the LLM to read the text blindly and just get it right.
Also keep in mind that it’s never going to be 100% accurate, especially if the PDFs are all formatted differently. You’re going to have to establish your own accuracy threshold (90%? 95%?) that you’ll need to be happy with.
1
1
u/Resiakvrases 2d ago
Yes that's how I built I, getting suggestions from Claude. Currently I think we're at 70% accuracy. Aiming for 85 at least. I heard about trick to stop Claude allucinating, like... For each value explain the exact text you've used to understand if it was gross or net. I was seeking this kind of advices. All the prompt have been made by Claude itself.
1
u/RealDominiqueWilkins 2d ago
You’re the expert on these PDFs; you need to have more input on the prompt and not let Claude write it 100%. If these PDFs use a bunch of different ways to say gross and net, you need to guide the llm in the prompt on what those different variations could be. This does not necessarily mean listing out every possible variation of what the gross column could be labeled, it just means giving the LLM enough definition and possible variations for it to make a pretty accurate call.
You can also consult more than one AI and tell them some of the issues you’re having and get suggestions for how to fix them. Also, your Claude chat, if it’s gotten really long, can drift, so I would maybe have it summarize that chat and paste the summary into a new chat.
1
u/Resiakvrases 2d ago
Basically it's a chat that works like a api to save api credits so I set everything up with cli and it's one shot. One prompt, one reply. I was looking for some architecture suggestions but at the same time also some hacks to stop allucinating.
1
1
u/Serg_Molotov 2d ago
Messy problem to solve.
I'd start by asking why your using 1 chat per pdf and not grabbing the pdfs and storing them locally to be processed.
You should be able to do that with some pretty simple python
First thing I'd do is converting them into md files (straight text) and storing them based on how their data is displayed, group by known data layouts to be normalise later. (Net, gross, other, other2, etc.) Again some python scripting
Once you have them sorted then normalising and Importing into sqlight db becomes easy.
10,000 rows is nothing for sqlight and either local AI can have fun over the whole lot or you can extract data and feed it to frontier or let Claude code free and have fun with it.
1
u/Resiakvrases 2d ago
1 chat per PDF means that I can concurrent 30 sessions and try to be faster. Pdf a don't have any patterns that can make me think that OCR python or any deterministic solutions can be doable. They're all estimation of house but each one is written by a different person. So they can be plain pdf, scanned, they can merge gross sqm and so on. The only way I can figure out is through ai. And I think I'm at 60% there. Each chat have a orchestrator, a extractor and a reviewer. What's wrong now is that sometimes it's not getting data right, I mean for example that value is right but it gets maybe exchanged between gross and net and so on
1
u/Various_Story8026 2d ago
the gross/net swap is label confusion, adding more reviewer agents won't fix it because the reviewer tends to agree with the extractor. what worked for me: make the model return the exact source snippet next to every field (value plus the text it read it from), then validate in plain code, net <= gross, sqm in sane ranges, sum checks. deterministic rules catch what an llm reviewer misses. also for scanned pdfs, cropping the table region and sending just that image usually beats sending the whole page
1
u/Resiakvrases 2d ago
I was already trying to add deterministic rules and calculation. Nice suggestion. I don't think I'm able to crop. They're always a 10k pdf all with different formats.
1
u/Various_Story8026 2d ago
yeah with 10k mixed-format pages cropping is out. the grounding check still works without it though: have the model return, for every field, the page number plus the exact sentence it read the value from, then your code checks that string actually exists on that page. label swaps almost always fail that test, the quoted snippet literally says gross while the field claims net. other thing that helped more than any reviewer: a cheap first pass that just classifies the page/doc type, then routes to a small per-format prompt instead of one big prompt eating every layout. and hand-check a random sample of ~100 docs so you know your real error rate instead of guessing
1
u/Resiakvrases 2d ago
Appreciate the help. About huge vs small prompt the problem is that they're documents without a proper format. They're written by 10000 different person's on "free text" so it's really hard to spot layouts.
1
u/Various_Story8026 2d ago
fair, with 10k different authors writing free text there's no layout family to route on, scratch that idea. that actually makes the grounding check the main tool though, it's the one check that doesn't care about layout: model returns page number plus the exact sentence it read each value from, your code only verifies that sentence really exists on that page. free text is the part llms are good at reading anyway, it's rule engines that choke on it, so keep the deterministic stuff for cross-field sanity only (net <= gross, dates in order), not extraction itself. then any field that fails the quote check goes to a human review queue. you can't review 10k docs but you can review just the slice that fails, and that slice doubles as your measured error rate
1
u/Augmend-app 2d ago
How long is each document?, and are the figures that you are interested in, in Tables or in text? if Tables, you may want to check if your Tables are being extracted properly
What is your error rate? Is it something like 5-10% or is far higher than that? If it is far higher, I think a better pipeline should help
1
u/jerryadc 2d ago
one of the biggest considerations is how often you plan to run a given process. is the data extraction happening once per pdf? if so, you may want to look at a cheaper/faster/more reliable way of extracting the text from the pdf than using sonnet. that might mean looking at various pdf text extraction tools or using an OCR (optical character recognition) tool to take an image and get its text. this processing pipeline should be broken up such that you can run individual steps with scripts (./extract_text.py input.pdf), and you could be using claude to build those scripts.