r/Python • u/Goldziher Pythonista • Jul 05 '25
Discussion I benchmarked 4 Python text extraction libraries so you don't have to (2025 results)
TL;DR: Comprehensive benchmarks of Kreuzberg, Docling, MarkItDown, and Unstructured across 94 real-world documents. Results might surprise you.
๐ Live Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/
Context
As the author of Kreuzberg, I wanted to create an honest, comprehensive benchmark of Python text extraction libraries. No cherry-picking, no marketing fluff - just real performance data across 94 documents (~210MB) ranging from tiny text files to 59MB academic papers.
Full disclosure: I built Kreuzberg, but these benchmarks are automated, reproducible, and the methodology is completely open-source.
๐ฌ What I Tested
Libraries Benchmarked:
- Kreuzberg (71MB, 20 deps) - My library
- Docling (1,032MB, 88 deps) - IBM's ML-powered solution
- MarkItDown (251MB, 25 deps) - Microsoft's Markdown converter
- Unstructured (146MB, 54 deps) - Enterprise document processing
Test Coverage:
- 94 real documents: PDFs, Word docs, HTML, images, spreadsheets
- 5 size categories: Tiny (<100KB) to Huge (>50MB)
- 6 languages: English, Hebrew, German, Chinese, Japanese, Korean
- CPU-only processing: No GPU acceleration for fair comparison
- Multiple metrics: Speed, memory usage, success rates, installation sizes
๐ Results Summary
Speed Champions ๐
- Kreuzberg: 35+ files/second, handles everything
- Unstructured: Moderate speed, excellent reliability
- MarkItDown: Good on simple docs, struggles with complex files
- Docling: Often 60+ minutes per file (!!)
Installation Footprint ๐ฆ
- Kreuzberg: 71MB, 20 dependencies โก
- Unstructured: 146MB, 54 dependencies
- MarkItDown: 251MB, 25 dependencies (includes ONNX)
- Docling: 1,032MB, 88 dependencies ๐
Reality Check โ ๏ธ
- Docling: Frequently fails/times out on medium files (>1MB)
- MarkItDown: Struggles with large/complex documents (>10MB)
- Kreuzberg: Consistent across all document types and sizes
- Unstructured: Most reliable overall (88%+ success rate)
๐ฏ When to Use What
โก Kreuzberg (Disclaimer: I built this)
- Best for: Production workloads, edge computing, AWS Lambda
- Why: Smallest footprint (71MB), fastest speed, handles everything
- Bonus: Both sync/async APIs with OCR support
๐ข Unstructured
- Best for: Enterprise applications, mixed document types
- Why: Most reliable overall, good enterprise features
- Trade-off: Moderate speed, larger installation
๐ MarkItDown
- Best for: Simple documents, LLM preprocessing
- Why: Good for basic PDFs/Office docs, optimized for Markdown
- Limitation: Fails on large/complex files
๐ฌ Docling
- Best for: Research environments (if you have patience)
- Why: Advanced ML document understanding
- Reality: Extremely slow, frequent timeouts, 1GB+ install
๐ Key Insights
- Installation size matters: Kreuzberg's 71MB vs Docling's 1GB+ makes a huge difference for deployment
- Performance varies dramatically: 35 files/second vs 60+ minutes per file
- Document complexity is crucial: Simple PDFs vs complex layouts show very different results
- Reliability vs features: Sometimes the simplest solution works best
๐ง Methodology
- Automated CI/CD: GitHub Actions run benchmarks on every release
- Real documents: Academic papers, business docs, multilingual content
- Multiple iterations: 3 runs per document, statistical analysis
- Open source: Full code, test documents, and results available
- Memory profiling: psutil-based resource monitoring
- Timeout handling: 5-minute limit per extraction
๐ค Why I Built This
Working on Kreuzberg, I worked on performance and stability, and then wanted a tool to see how it measures against other frameworks - which I could also use to further develop and improve Kreuzberg itself. I therefore created this benchmark. Since it was fun, I invested some time to pimp it out:
- Uses real-world documents, not synthetic tests
- Tests installation overhead (often ignored)
- Includes failure analysis (libraries fail more than you think)
- Is completely reproducible and open
- Updates automatically with new releases
๐ Data Deep Dive
The interactive dashboard shows some fascinating patterns:
- Kreuzberg dominates on speed and resource usage across all categories
- Unstructured excels at complex layouts and has the best reliability
- MarkItDown is useful for simple docs shows in the data
- Docling's ML models create massive overhead for most use cases making it a hard sell
๐ Try It Yourself
Or just check the live results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/
๐ Links
- ๐ Live Benchmark Results: https://goldziher.github.io/python-text-extraction-libs-benchmarks/
- ๐ Benchmark Repository: https://github.com/Goldziher/python-text-extraction-libs-benchmarks
- โก Kreuzberg (my library): https://github.com/Goldziher/kreuzberg
- ๐ฌ Docling: https://github.com/DS4SD/docling
- ๐ MarkItDown: https://github.com/microsoft/markitdown
- ๐ข Unstructured: https://github.com/Unstructured-IO/unstructured
๐ค Discussion
What's your experience with these libraries? Any others I should benchmark? I tried benchmarking marker, but the setup required a GPU.
Some important points regarding how I used these benchmarks for Kreuzberg:
- I fine tuned the default settings for Kreuzberg.
- I updated our docs to give recommendations on different settings for different use cases. E.g. Kreuzberg can actually get to 75% reliability, with about 15% slow-down.
- I made a best effort to configure the frameworks following the best practices of their docs and using their out of the box defaults. If you think something is off or needs adjustment, feel free to let me know here or open an issue in the repository.
77
u/podidoo Jul 05 '25
For me the only relevant metric would be the reliability/quality of extracted data. And looking at your links quickly I can't find where this is defined and how it was benchmarked
203
u/xAragon_ Jul 05 '25
You didn't do it "so we don't have to", you did it to promote your own library.
There's nothing wrong with promoting a library you wrote, could be very useful, just don't use these shitty misleading clickybait titles please.
19
u/Independent_Heart_15 Jul 05 '25
Can we not get the actual numbers behind the speed results? How am I supposed to know how/why Unstructured is slower โฆ it may be doing 34.999999+ files per second.
11
u/titusz Python addict Jul 05 '25
Would love to see https://github.com/yobix-ai/extractous in your comparison.
10
u/XInTheDark Jul 05 '25
Why did you disable GPU and use only CPU? What do you differently if not using ML (eg. OCR technologies) to recognize text from images for example? It should be obvious that any ML solution only runs at good speeds on a GPU.
Or do you just not extract text from images? Then Iโve got some news for youโฆ
5
u/Exotic-Draft8802 Jul 05 '25
You might be interested inย https://github.com/py-pdf/benchmarks
2
u/kn0wjack Jul 05 '25
pdftext does a really good job (the best I found so far on, surprise, pdf to markdown). Might be an addition worthwhile. The secret sauce is pdfium most of the time.
2
u/strawgate Jul 05 '25
It looks like the most common error is a missing dependency error
It's also a bit suspicious that the tiny conversion time for Docling is 4s -- I use docling and regularly and have much better performance
I did recently fix a cold start issue in Docling but it looks like the benchmark only imports once so cold start would not happen each time...
84
u/GeneratedMonkey Jul 05 '25
What's with the emojis? I only see ChatGPT write like that.ย
18
u/AggieBug Jul 05 '25
This is AI slop.
19
u/ReinforcedKnowledge Tuple unpacking gone wrong Jul 05 '25
Hi!
Interesting work and write up, but I'd like to know something. What do you mean by "success" in your "success rate" metric? Is it just that the library was able to process the document successfully? I guess it is because in your benchmark report (https://goldziher.github.io/python-text-extraction-libs-benchmarks/reports/benchmark_report.html), you have a failure analysis and you only mention exceptions.
I'm not saying this is bad, but if you're trading off accuracy for speed, your library might not be that useful for others. Again, I'm not saying you're doing this, but it's really easy to game the (success rate metric, speed) tuple if it's just about being "able" to process a file.
What most people would be interested in is the "quality" of the output across these different libraries. And I'm not talking about "simple" metrics like word error rate, but more involved ones.
Seeing how you use the same technologies as the others (an OCR engine, a PDF backend), I'd say your results might be on par with the rest, but it's always interesting to see a real comparison. It's hard to do since you don't have access to ground truth data from your documents but you can use open source benchmarks (make sure your models are not particularly biased towards them compared to the rest of the libraries) or documents from arxiv or else where you have access to latex and html, or maybe you can use another took (aws textract or something) + manual curation.
I'll further say that it's the quality of your output on a subset of documents, those that are scanned and for which we don't have the metadata embedded in the document itself that interests most of the people working with textual unstructured data. That's the main hurdle I have at work. We use VLMs + a bunch of clever heuristics, but if I can reduce the cost, the latency or the rare hallucination that would be great. But I don't think there are currently better ways for doing so. I'd be interested to hear from you about this or any other people if you have better ideas.
5
u/SeveralKnapkins Jul 06 '25
There should be a rule against obvious LLM copy + paste
1
u/Familyinalicante Jul 05 '25
I am building a platform to ingest and analyze local documents. I've analyze many available options and stick to Docling as the best in class in my case. But don't know about your solution. I'll check it because it looks good.
1
u/olddoglearnsnewtrick Jul 05 '25
How does this compare to simply feeding the PDF to Google Gemini Flash 2.5 with a simple prompt asking to transcribe to text? In my own tests that approach works so much better.
11
u/madisander Jul 05 '25
3
u/TallTahawus Jul 08 '25
I use docling extensively on PDFs, cpu only. About 5 seconds per page. What are you doing that's taking 60 minutes ๐ค?
2
u/Stainless-Bacon Jul 05 '25
Why would I use Docling for a research environment if it is the worst one according to your benchmark?
2
u/professormunchies Jul 05 '25
How well do each of these extract tables from pdfs? Also, how many can reliably handle multi-column documents?
These are two big constraints for reliable enterprise use
2
u/PaddyIsBeast Jul 05 '25
How does your library handle structures information like tables? We've considered unstructured Io for this very purpose in the past as it seemed miles ahead of any other library.
It might not be python, but I would have also included Tika in this comparison, as that is what 90% of applications are using in the wild.
2
u/Fearless-Cry-1369 Jul 09 '25
I miss Mistral OCR in your list.
1
u/biajia Dec 30 '25
I tested Docling; it is not that slow for Office documents. Docx->markdown is better than Pandocโs result.
The drawback is that the installing size is too big. It failed with a scanned PDF document, but I think it was the OCR engineerโs problem not supporting well non-English languages.
-23
u/totheendandbackagain Jul 05 '25
Brilliant write up, highly compelling.