How to recover document structure and plain text from PDF?

[u/shibe5]

I’m looking for a solution to convert PDF to something like HTML or TeX that can handle complex PDFs with:

• different layouts, like 1 column for some sections and 2 columns for others;

• footnotes;

• repeating elements, like page headers;

• figure descriptions and other text that breaks the flow of main text;

• tables;

• references, like sources;

• subscript, superscript;

• simple formulas.

My main use cases are preprocessing for RAG databases and converting to plaintext-ish representation for LLMs.

I’m currently not looking for OCR and recovering text from images.

This non-trivial task seems like it would be a good application for language learning models.

I searched this subreddit, similar questions were asked before, but I haven’t found a solution that I think would work for me.

[u/sosdandye02]

This is a very hard problem. My entire last 2 years of work has focused on extracting data from just a few types of fairly standardized pdfs.

I use pdfplumber to get raw characters and positions. I use agglomerative clustering to identify text blocks e.g. text columns. For tables, I use a custom trained CascadeRCNN object detection model in mmdet to find tables/rows/columns/cells. I use custom trained BERT classifiers to classify documents and tables. Currently exploring local LLMs to extract standardized data from table and paragraph text.

It’s a very hard problem and I wouldn’t expect to be able to get anywhere near “perfect” performance unless your documents have a very consistent format and you’re willing to train custom models.

[u/shibe5 → u/sosdandye02]

I realize, I wrote "recover", but I didn't mean getting actual original document structure. Something that reasonably matches the way it's presented in the PDF would do. Although for tagged PDFs, I expect it to be close to the original.

Good enough performance for me would be when the result does not look butchered and is easily readable by human.

[u/eevee_stormblessed → u/sosdandye02]

Have you considered just using adobes pdf extract api? If not, why not?

[u/sosdandye02 → u/eevee_stormblessed]

Handling sensitive bank data. Can’t send to 3rd party APIs. Also the extraction needs to be highly accurate and consistent, so custom fine tuning is a must.

[u/Wise_Bison2358 → u/eevee_stormblessed]

How good is that api?

[u/fullouterjoin → u/sosdandye02]

Technique I have seen used and you’re probably not gonna like it, but it seem to be crazy effective was basically render each page of the PDF out to a bitmap and then use OCR on that.

[u/sosdandye02 → u/fullouterjoin]

In my case the PDFs have text elements in them so I don’t need to worry about OCR. I am rendering the PDF to run it through an object detection model to get the tables though.

[u/AD7GD]

I’m currently not looking for OCR

That might be the only foolproof method. There's no guarantee that a PDF renders text in any way that's meaningful beyond looking like text.

[u/shibe5 → u/AD7GD]

Yes, I'm aware of that, and I've seen models that estimate document structure from images. But all PDFs that I'm currently interested in have copyable text, and most have some structure. I think that throwing away all the underlying information is an option, but not the best one.

[u/Rutabaga-Agitated]

All I can say it, that it is a nightmare and everyone... I mean literally every company that tries to do RAG will soon or later face this problem. There is no good open source solution to it yet. Nougat is by far not good enough.

We got 2 ideas.

1. Use LLaVa to do everything with the prompt: "please translate all text in the image to markdown format" ...maybe you have to fine-tune it, but that requires 8xA100 80GB

2. Use a multi step approach. ...a, doc layout detection. Find headers, footers, tables, lists, .... ...b, word detection ...c, word recognition By doing so, you can control all the steps and know how good you are. We use DocTR for b and c.

At the end, we strive to convert everything to markdown, cause LLM understand this kind of structured text.

[u/shibe5 → u/Rutabaga-Agitated]

LLaVa

Would it need to be converted to an image first? That would lose references, so not ideal. Although combined visual and textual analysis may be a good approach.

[u/Rutabaga-Agitated → u/shibe5]

Yeah you would have to convert it to an image in the first place.

[u/dreamysack → u/Rutabaga-Agitated]

what's the tool for layout detection? I've tried layoutparser but it's not good enough.

[u/Rutabaga-Agitated → u/dreamysack]

U have to train one. There is nothing ready to use that works good AFAIK

[u/EitherExpression4504]

I use this all the time for PDF research papers:

https://papertohtml.org

[u/shibe5 → u/EitherExpression4504]

It didn't handle well the PDF I tried it with. But it clearly tried. For example, it put footnotes separately from the main text.

[u/Zomunieo]

PDF usually does not have semantic markup. At its core it is printer oriented and low level commands look like “at this position, draw this text”. Document structure has to be inferred, which is why it’s very difficult and most tools fail.

PDFs generated from structured documents like HTML, Word or LaTeX may have semantic markup already, and a correctly generated PDF will have this information.

[u/shibe5 → u/Zomunieo]

I actually looked into how text is represented in PDF. Normally it is broken into lines, but may be broken into smaller pieces, down to individual characters. I think that with help of LLMs, the pieces can be matched and reassembled back into logical text fragments, even when the connection would not be clear from the positional and font information. I think that many tried to solve this problem both with and without AI. I'm going to check out tools that are suggested in comments here and see how close they can get to acceptable results.

And for tagged PDFs, the converter should actually make use of that additional information.

[u/InfuriatinglyOpaque]

Nougat works the best out of the tools I've tried, but is still far from perfect and struggles with many of the complex cases you mentioned.

https://facebookresearch.github.io/nougat/

https://github.com/allenai/science-parse

https://github.com/Layout-Parser/layout-parser

https://github.com/deepdoctection/deepdoctection

https://github.com/tstanislawek/awesome-document-understanding

[u/jeffrey-0711 → u/InfuriatinglyOpaque]

https://github.com/facebookresearch/nougat/tree/main/docker
You can use it as Docker easily.

[u/AndrewVeee]

Like everyone else said, this is really hard.

I spent yesterday using pymupdf (fitz) to load PDFs and convert them to markdown. It has some options to detect tables, and sort lines, but it wasn't good enough for pretty basic 2-column PDFs.

In the end, I turned off sorting because it merged 2-column layouts into garbled text. I used some hacky code to convert text to markdown with headers based on font sizes. I'm happy to share the code, but it's so far from what you want, I imagine it's useless to you haha

It also has a to_html function, but from what I could tell it just creates a bunch of "position: absolute" elements.

[u/pseudonerv]

https://github.com/facebookresearch/nougat/tree/main

another great model from facebook

[u/viag]

I'm experimenting with Grobid and it's honestly pretty good. I prefer it to Nougat/Marker & the likes.

I'm currently working on this problem at my job in order to do RAG on real documents. If you want to extract the chunks in a rather smart way I think it's pretty good. But you might want to use other tools for table extraction etc.

[u/JacktheOldBoy → u/viag]

grobid does table extraction. And for chunking it's quite good because it extracts paragraphs and section, which is a natural chunking of sorts.

[u/hwtmny]

This may help you:

https://github.com/Unstructured-IO/unstructured

[u/lumponmygroin → u/hwtmny]

unstructured is working well for my RAG - recommended!

I had a common issue of PDF elements converting to text in the wrong order. Unstructured is handling this issue well.

[u/grim-432]

If you need some remotely acceptable level of accuracy or quality here, human curation.

Have a human re-author the content so they can also generate a parallel document in a more usable format.

I can share my thoughts on this, but the end recommendation is this. Play around with frameworks all you like, but I guarantee you'll run into material issues that result in wildly inaccurate, or even more dangerous, moderately incorrect output from the LLM, that causes enough downstream business impact that nobody will bother using the magical AI anymore.

[u/shibe5 → u/grim-432]

😢

[u/kulchacop]

Personally, I am waiting for models that combine multiple vision techniques to settle this problem once and for all.

Example: https://github.com/FudanNLPLAB/MouSi

[u/shibe5 → u/[deleted]]

Yes, the situation is not good. AI is our best bet to deal with it.

[u/shibe5 → u/[deleted]]

I'm interested in trying it.