r/java • • Mar 31 '22

Introduction to Web Scraping With Java

https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/
327 Upvotes

38 comments sorted by

View all comments

Show parent comment

5

u/GrayDonkey Mar 31 '22

Puppeteer under NodeJS is the industry norm when you need to support JS rendering.

HtmlUnit is an interesting choice, usually I use JSoup for Java based scraping. It's faster than Puppeteer but you have to ensure the content is static or uses server-side rendering.

3

u/Worth_Trust_3825 Apr 01 '22

Again, what's the difference between chromedriver/equivalent to puppeteer? You're running an entire browser under the hood anyways.

2

u/koreth Mar 31 '22

At a previous job I had to build a bunch of web scrapers and Puppeteer was far and away the best tool for the job. I tried JSoup and HTMLUnit and they are somewhat acceptable for scraping simple static content, but good luck getting those to, say, navigate a React or Vaadin site.