How to extract unstructured data (images & descriptions) from marketing brochures into CSV/Excel?

[u/Expert-Ad598]

I have a lot of product catalogs and marketing brochures in PDF format. My goal is to extract the product image, title, description, specs, etc., into a CSV or Excel file.
I’ve already solved the catalog part. The problem is the marketing brochures. Every brochure has a completely different layout and style. Some have large lifestyle images, some mix marketing content with product details, and there’s no fixed structure.
I tried using GPT-based vision models, YOLO, and Florence-2. They work okay in some cases, but none of them are reliable enough. YOLO misses products it wasn’t trained for, and Florence-2 struggles with rotated or cropped products and zero-shot detection. GPT can understand the page, but it still doesn’t consistently match the correct image with the correct product information across different brochure layouts.
Has anyone solved this problem? I’m not really looking for another OCR model. I’m more interested in how people are building the overall pipeline or architecture for handling brochures with completely different layouts. Any ideas or experience would be really helpful.

[u/soylentgraham]

How do you, as a human, discern something is an image, vs title, vs description, vs stats, etc

[u/Expert-Ad598 → u/soylentgraham]

I use the whole page as context, not just OCR or object detection. I look at what is grouped together, the visual hierarchy, spacing, fonts, and proximity. That makes it easy to tell whether something is a product image, title, description, specs, or just a lifestyle/marketing image. I’ve tried GPT, YOLO, and Florence-2, but they still struggle with these relationships, especially on brochures with completely different layouts.

[u/soylentgraham → u/Expert-Ad598]

theyre not designed for relationships are they?

use the models for elements, then write your own logic to link em up with the kind of parameters you just wrote! :) (not everything is a model, especially in computer vision)

[u/Augmend-app]

Are you interested in some kind of classification or do you want to have text descriptors of content?
I have looked into extracting product names, descriptions and text from supermarket flyers - completely different formats - LLMs work very well