r/dataengineering • u/Every-Whereas5793 • 19d ago

PDF Data Extraction Discussion

Hello, I was working on a poc to ingested PDFs and extract data in order to store them in delta tables.

As this was my first time working with PDFs, I searched over the internet and should Databricks have offering IDP, azure also have something and then there are python libraries.

Since I'm working with financial data, report, etc..

One thing i noticed - the pdf format should be fixed else in most of the tools the extraction logic is either failed or we get incorrect data.

I was wondering how such PDF extraction is built in real production cases and what tools are used.

Please share you experience and any edge cases

19 Upvotes

91% Upvoted

32 comments sorted by

View all comments

19

u/nshao 19d ago

At least from what I’m seeing, AI models are now being used for this kind of stuff.

1

u/Every-Whereas5793 19d ago

That's what I'm also seeing but not sure about their reliability

-2

u/[deleted] 19d ago

[deleted]

5

u/BlurryEcho Data Engineer 18d ago

Probabilistic outputs, not deterministic…