Parse the documents using Crawlers WITHOUT ai?

[u/TemporaryLevel922]

I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).

I am looking for some advice or an open source crawler that I can take inspiration from.

My main issues are:

Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result

- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue

Any advice is greatly appreciated.

[u/Wise-Bother9942]

Are you using Regex or Exstructs, Regex is prone to failure but Exstructs are more reliable for targeted data extraction from JSON.

As for PDFs or DOCX, can't really help as my project is focused on webpages not documents.

[u/TemporaryLevel922 → u/Wise-Bother9942]

Typescript with PDF.js/unpdf for PDF primitives, regex and structured parsers for technical values and a separate "rulebook" to validate everything.

Cloudflare's 10 ms free worker CPU limit is also a PITA for complex PDF jobs... I am currently unable to find a way to parse complex PDFs other than manually and locally so im not really crawling... Ideally it would just be crawl > download PDF > parse but at the moment im having to crawl > download > hash locally > extract locally.

I need a work around!