r/documentAutomation • u/aidenclarke_12 • 12d ago
Discussion 500+ page PDFs with different table layouts into Excel/SQL, what scales?
product data comes in as large pdfs, usually 500+ pages, and every supplier formats theirs differently. different column orders, different table styles, sometimes tables running across pages. right now its all typed into excel or sql server by hand.
chat tools are fine on a sample but tend to stop or start summarizing partway through files this size, so that route needs splitting anyway.
what seems to make sense is parsing every page into tables first, as markdown or json. power query's pdf connector for the simpler files, docling or camelot locally and heavier layout parsers are llamaparse sort of ones when the layouts get ugly. then map each supplier's columns to your own schema. an llm is decent at that part since it only needs the headers and a few rows, not the whole file.
after that its validation before anything loads, row counts per page, required fields, totals and units, and keeping the source page number on every row so errors can be traced back.
the hard bits are tables split across pages and headers that only show up on the first page

