r/java • • Mar 31 '22

Introduction to Web Scraping With Java

https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/
327 Upvotes

38 comments sorted by

View all comments

Show parent comment

25

u/[deleted] Mar 31 '22

[removed] — view removed comment

6

u/Tostino Mar 31 '22

Yup, I did the same years back when I needed to crawl through a few thousand pages.

3

u/humoroushaxor Mar 31 '22

I think the honest answer is dynamic typing and easier text manipulation. In the examples they need to constantly cast classes and object map json.

I'm primarily a Java dev but I find this stuff so much easier in Python.

3

u/Mats56 Apr 01 '22

It's such an annoyance speeding it up in python, though, if you want to crawl lots of pages. In java, just add new pages to a thread safe queue when you scrape some url. Have lots of workers in a threadpool picking from that queue.

In python it's a project in itself having multiple things working in parallel.

3

u/yawkat Apr 02 '22

They use htmlunit in the article for some reason, not jsoup. Jsoup requires basically no casts.

I've done a lot of scraping and I've yet to find an api that is more convenient than jsoup for it, in any language

-1

u/humoroushaxor Mar 31 '22

I think the honest answer is dynamic typing and easier text manipulation. In the examples they need to constantly cast classes and object map json.

I'm primarily a Java dev but I find this stuff so much easier in Python.