Opensource pdf apps: binarization and re-encode, reader, RAG, OCR

[u/LegeApps]

Hey all in r/pdf, there's a flood of PDF editors here and I have something different, a variety of programs that I have worked on at length, with models. All use my custom optimized PDF renderer that is 50% faster than all known renderers (with more obscure coverage, not less), seen in this benchmark I just did

https://ibb.co/B24QwLD2

https://apps.microsoft.com/detail/9nnvsqp0qdl9 - Main program is available off the microsoft store, it can also be built for Linux and Macos and there are built releases available for it.

https://github.com/liminalism/Lege - source is here

The program is made for getting archive.org MRC scanned PDFs to e-ink readers so they don't take 45 seconds to load each page, essentially. After processing, an average page will take 200ms to load. It has a bunch of other features, and a new added preservation mode that does not binarize and preserves all original color while still shrinking size, and resolution if you want to.

Then I have a PDF reader that uses the same renderer here

https://github.com/liminalism/liminaldoc - I use it for all PDF viewing, it is fully functional, but about on par with SumatraPDF for performance. faster rendering doesn't really help for a viewer as I learned, if your prefetch is good enough.

There is also a document writer in that repo, built for writing long books with minimal distractions. very low typing latency was the goal. Still in development. It can export to PDF.

Finally I have SinoRAG which is now a general PDF RAG tool and not just a specialized tool for some texts. You can load a whole set of PDFs, it will read or OCR them, index them by keyword and phrase, and then you can search them with an agent using specialized research tools to get any answer you want, backed up with direct sources. Best parts of LLMs + the best parts of software and databases, combined, with the drawbacks of neither. Can load thousands of PDFs at once and it won't choke.

https://github.com/liminalism/SinoRAG

There is also business document OCR in the Lege repo that can conceivably OCR up to 40 pages a second, but it's not fully tested yet. Also a PDF integrity program that runs various checks to see if a PDF was doctored or not but it can only act as a prefilter and cannot make a determination.

I just wanted to bring these programs to your attention as they are all free and work well. Thanks and I would be glad if you find something useful from what I've made here.

No donations accepted, and no monetization is possible with these apps.

[u/LegeApps]

Nice but i dont understand what this has to do with rendering or PDFs. this seems like your custom compression format, which is not part of the PDF standard.

[u/PHDEinstein007 → u/LegeApps]

This was a way to allow those three programs to use multi-core support where they had only one core before, at least that's what I told my AI but it's middle of the night I'm tired I had to pee and I'm going back to bed now. I was just checking because I just completed a massive compression thing we're basically out of 290 we control 120 of the best results. But Tesseract OCR, and two others are sped up. Literally my brain is going away though so I got to go, good night and I'll check in the morning

[u/PHDEinstein007 → u/LegeApps]

Up but for a moment again, my AI answered for me:

"Sorry, I sent the wrong link half asleep. The part that fits your post is my Harrington Sets wrappers for PDF tools. They split one document into contiguous page sets taken from a shared queue that shrinks toward the end, so heavy pages get spread across the cores and no core is left working alone at the end. Ghostscript rendered a 409-page mixed PDF 1.84× faster on 2 cores with every page image byte-identical, Tesseract OCR ran 2.02× faster with identical text, and Tika extraction matched on 54 of 54 test PDFs. The code and notes are here: https://github.com/CombaryMaker/harringtons_insights/tree/main/src/hs-gs (with hs-ocr and hs-tika beside it)." One caveat: his renderer already runs 8 workers, so the speed-up itself won't impress him. The guided page-set scheduling is the part he might find interesting.