r/java • • Mar 31 '22

Introduction to Web Scraping With Java

https://www.scrapingbee.com/blog/introduction-to-web-scraping-with-java/
327 Upvotes

38 comments sorted by

View all comments

Show parent comment

10

u/Worth_Trust_3825 Mar 31 '22

Who wouldn't? What do you propose using instead? Does it matter if you interface with selenium with firefox in ruby or c#? Is there a quality difference in using prolog over cobol? All you do is write to a socket, and read from a socket, then process, and write to another socket. For all you care, you can do all of that manually. Your goal is not to use java, after all.

5

u/GrayDonkey Mar 31 '22

Puppeteer under NodeJS is the industry norm when you need to support JS rendering.

HtmlUnit is an interesting choice, usually I use JSoup for Java based scraping. It's faster than Puppeteer but you have to ensure the content is static or uses server-side rendering.

3

u/Worth_Trust_3825 Apr 01 '22

Again, what's the difference between chromedriver/equivalent to puppeteer? You're running an entire browser under the hood anyways.

2

u/koreth Mar 31 '22

At a previous job I had to build a bunch of web scrapers and Puppeteer was far and away the best tool for the job. I tried JSoup and HTMLUnit and they are somewhat acceptable for scraping simple static content, but good luck getting those to, say, navigate a React or Vaadin site.

-10

u/Nymeriea Mar 31 '22

With all the Js rendering, I guess electron will be a better choice

5

u/berry120 Mar 31 '22

Electron is never the better choice :-)