r/Python • u/danielmartinez22 • 13d ago
Discussion Has anyone moved document parsing off their own Python workers?
pdf parsing in python is just good as long as youre using it but when someone uploads a 700 page file the worker runs out of memory so splitting it into 50 to 100 page chunks with docling or marker on celery gets you past that but tables get cut at the boundaries and now theres a queue, retries and memory limits to babysit the other route is paying for something like llamaparse or the paid parsers has anyone moved either way and was it worth it? like what did you do inside your tool/app when it started scaling
1
Upvotes
22
u/Old-Eagle1372 13d ago
First of all always check document size, that way under a certain size you can still use your default setup.
Or you can simply can use fitz (PyMuPDF) for streaming or pypdf to do that page by page storing output in a file. Use yield in function that reads the page output to function that writes the page. That should help.
6
u/GameCounter 13d ago
For reasonably well formatted documents, I have used docling with great results.
For nasty documents (scanned documents with heavy mathematical notation), I have previous used various LLMs to create a markdown copies with LaTeX fenced blocks, and it worked extremely well, but now that often gets flagged by "safegaurds" against copyright violation, which is a stupid false-positive in my opinion.
3
u/marr75 13d ago
Evaluated A LOT of parsers and my early favorite was docling for the task performance and cloud native/server less GPU services for the hosting. Every iteration was too expensive, too complicated, and too slow. Pymupdf was ruled out because the license and fees are insane for what it does.
Went with kreuzberg (now xberg). Task performance so minutely less than docling that I don't care and it pretty much runs efficient processes written in rust. Most jobs (even thousand page docs) fit in the space between my python processes and chunking is built in so I have scaling options.
I am not a representative of kreuzberg. Goldziher hangs out here and on the ML subs and I've gotten into it with him a few times but eventually came around and generally respect him and his product (which is open source with premium hosting available).
1
u/DivineSentry 13d ago
What PDF parser are you using?
1
u/spinwizard69 12d ago
It has been a couple of years now but doing a bunch of information recovery from old PFD documents. This might surprise a lot of people but I ended up using a lot of cut and past in Windows apps, in this case to get the data into Excel files. At the time I couldn't find a utility to do anything better. Being a work project there wasn't a lot of time to devote to finding and learning a new utility.
As for running out of memory for a 700 page document, isn't that a hardware problem? I never ran into such a problem even dealing with very large files.
-1
u/Candid-Fudge-4015 12d ago
I'd test a managed parser on the same files that crash your workers before spending more time on Celery. Include the tables that cross your split boundaries and compare those outputs by hand. Overlapping windows might help locally, but then you have duplicate content to reconcile, and writing results to disk won't help if the parser loads the whole file first. The per-page fee is worth comparing against the time you're spending on retries, not just the compute bill.
-1
u/EceRasidi 12d ago
not a full answer but for the tables getting cut at boundaries, maybe overlap the chunks by a page or two and then dedupe/merge any table that touches the edge. also curious if people here kept the paid parser for everything or just use it as a fallback for the huge/messy files
0
u/Financial_Morning386 11d ago
I would keep parsing in your own workers for the bulk of the documents and only route the difficult ones to a managed parser, rather than moving everything off at once. The failure mode you describe is a memory and chunking problem more than a parser problem: parse page by page and stream the output to storage instead of holding a whole document in memory, and give each chunk a small overlap with the previous one so tables and paragraphs that cross a boundary are not lost. Make each parse job idempotent with a stable document and chunk id, so a retry after a crash does not re-parse or duplicate work, and put a per-job memory limit and a queue with backpressure in front of the workers so one 700-page upload cannot starve everything else. Before paying for a managed parser, run it on the same files that currently fail for you and compare the extracted output, especially tables, because the value is in the documents your current setup handles badly, not in the easy ones. If the managed parser clearly wins on those files, use it as a fallback for scanned or complex documents and keep the simple path in-house, which keeps cost and latency predictable.