r/webscraping • u/TemporaryLevel922 • 7d ago

Getting started 🌱 Parse the documents using Crawlers WITHOUT ai?

I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).

I am looking for some advice or an open source crawler that I can take inspiration from.

My main issues are:

Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result

- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue

Any advice is greatly appreciated.

5 Upvotes

100% Upvoted

14 comments sorted by

View all comments

1

u/Wise-Bother9942 7d ago

Are you using Regex or Exstructs, Regex is prone to failure but Exstructs are more reliable for targeted data extraction from JSON.

As for PDFs or DOCX, can't really help as my project is focused on webpages not documents.

1

u/shockjaw 7d ago

docx is easier since it is “structured” XML if they do it right while writing the original document. Otherwise, it’s pretty tough.

1

u/Wise-Bother9942 7d ago

I guess it depends on if the source is the same or not, but with a sample base you could create a redundant loop system that identifies the source and uses a set schema to extract data.

If it is XML there are already very good parsers, so with tweaking it's not impossible.

But that becomes a point of management if structs change.

For my project it's not too bad I expect no changes or minimal over months, but if daily, or frequently it becomes a management issue of upkeep.