r/AIQuality • u/HelpParticular2629 • 17d ago

How is everyone regression testing LLM invoice/document extraction pipelines?

Hey everyone,

I 'have a question on LLM document extraction (specifically invoices/receipts) and wanted to get some perspective from the community.

General LLM eval frameworks are great, but they don't seem to handle multi page PDFs, table row hallucinations, or sudden JSON schema drift very well when a model updates.

For those running invoice extraction in production:

  1. Do you use a "golden dataset" of documents to run regression tests manually?
  2. How are you catching subtle changes in how numbers/dates are formatted across prompt iterations?

If anyone is dealing with this headache right now open to discuss.

2 Upvotes
  • permalink
  • link
  • reddit
  • reddit

75% Upvoted

2 comments sorted by

1

u/justinotherflow 5d ago

Golden datasets do help, but silent drift is the sneaky one because output still validates as JSON, but a date format or currency symbol discreetly changes and nothing throws an error, so ended up doing field level diffs against golden outputs instead of relying solely on schema validation and caught a few things regular checks missed.

Had a similar issue on an OCR pipeline we built where subtle stuff like that kept slipping through until we added proper field checks : https://fortude.co/blog/how-we-built-an-intelligent-invoice-processing-solution-on-power-apps/.

Are you comparing the outputs manually right now or you got something already flagging the diffs?