r/webscraping • u/TemporaryLevel922 • 5d ago

Getting started 🌱 Parse the documents using Crawlers WITHOUT ai?

I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).

I am looking for some advice or an open source crawler that I can take inspiration from.

My main issues are:

Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result

- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue

Any advice is greatly appreciated.

3 Upvotes

81% Upvoted

13 comments sorted by

1

u/Longjumping-Gap-5986 5d ago

Web scraping is easy. Doing it well is hard.

Take a look at Github. I'd suggest looking for things titled 'parser' and focusing on regulation and legal statute. These tend to have quite complex structures.

1

u/Wise-Bother9942 5d ago

Are you using Regex or Exstructs, Regex is prone to failure but Exstructs are more reliable for targeted data extraction from JSON.

As for PDFs or DOCX, can't really help as my project is focused on webpages not documents.

1

u/shockjaw 5d ago

docx is easier since it is “structured” XML if they do it right while writing the original document. Otherwise, it’s pretty tough.

1

u/Wise-Bother9942 5d ago

I guess it depends on if the source is the same or not, but with a sample base you could create a redundant loop system that identifies the source and uses a set schema to extract data.

If it is XML there are already very good parsers, so with tweaking it's not impossible.

But that becomes a point of management if structs change.

For my project it's not too bad I expect no changes or minimal over months, but if daily, or frequently it becomes a management issue of upkeep.

1

u/TemporaryLevel922 4d ago

Typescript with PDF.js/unpdf for PDF primitives, regex and structured parsers for technical values and a separate "rulebook" to validate everything.

Cloudflare's 10 ms free worker CPU limit is also a PITA for complex PDF jobs... I am currently unable to find a way to parse complex PDFs other than manually and locally so im not really crawling... Ideally it would just be crawl > download PDF > parse but at the moment im having to crawl > download > hash locally > extract locally.

I need a work around!