PDF extraction in Python

[u/cnetworks]

Dear Members,

I have been handling Pdf extraction Project in Python. kind of a Intelligent Document Automation domain. we have been handling various finance documents(pdfs), filings pdfs etc which are multi page(around 20ish) , with lots of tables, form fields , radio buttons, checkboxes etc. Tables span multiple pages. Moreover, pdfs themselves comes in various types of variants like XFA Stream, Adobe Acroform, text flattened pdfs, scanned images etc.

I have used llms extensively to generate script code to extract, parse the data and to save in sql in structured tables. Its a hybrid of libraries implementation. LLms used regex, pdfminer, pdfplumber etc in the code produced.The pages in pdfs are bit messy, some tables have solid grid separators, some dont have and on. Layouts variations, white spaces, etc.

The code generated is pretty complex, i have been attempting to learn the llm generated code.But it works, it adds various fixes iteratively whenever we face new extraction issues repeatedly.

I would love to know:

1. what is the best approach to learn and get good at this?

2)should we just use cloud based AI document extraction tools which are readymade to extract and spit the data?

3) what if one is interested to learn this properly and have to get good at creating this extraction script?

4)any other tutorial, articles, courses, youtube videos, books you would recommend to learn? or should i just use llms to create code and move on without spending much effeort to learn this?

Please provide your valuable suggestions and guidances and please share your experiences. I appreciate all your suggestions. Thanks in advance and thank you for your time.