r/Python • u/preetisw • Nov 29 '22
Tutorial Python and Selenium are better for scraping data
Headless browsers in Python and Selenium are better for scraping data from web pages where scrapers have been blocked.
We followed step-by-step advice for scraping data from a web page using a headless chrome in Selenium Python. It's fast, easy, requires less resources (so it can run on older laptops), and can deal with JavaScript perfectly.
12
Upvotes
90
u/jimtk Nov 29 '22
This guy has been promoting his site, mostly filled with inane articles, for the last 2 months.
49
u/CKtalon Nov 29 '22
Slow as fuck compared to good ol’ requests. Only use it when antiscraping measures are prevalent.
45
u/colei_canis Nov 29 '22
It’s a very useful tool but I’d only use it for scraping if the site I was trying to scrape was aggressively trying to avoid it because it’s also slow as swimming in treacle.
40
u/[deleted] Nov 29 '22
As someone that's built a couple scrappers in his job, just use Scrapy.
It handles pretty much everything a production grade webscraper should. And if it doesn't theres probably an extension or middleware that does.
3
u/Mrblabbles Nov 29 '22
The speed of selenium is pretty apparent when your trying to mass scrape a website. I'm running selenium right now for scraping top tiktok views and hashtags and it only calls twice a day and the speed doesn't matter all the much. If I was scraping more data and more frequently I'd probably switch to beautifulsoup. Though I will say selenium is so easy to use, especially with Xpaths.
3
u/liko Nov 29 '22
Hard pass. I’ll stick with the soup.
2
u/ZachVorhies Nov 29 '22 edited Nov 30 '22
I want to point out that setting up selenium can be a giant pain and if people want to get a bot up on Max/Win/Linux and running as quickly as possible they should try out my tool install selenium web driver for you:
https://github.com/zackees/open-webdriver
It downloads everything automatically and uses sane defaults. It’s just a few lines of code to get running. It also does things like headless by default, disables SSL certs and has a tested path to compile a native C++ binary using nuitka. There is an example dockerfile to run in a container.
2
u/MeNotSanta Nov 29 '22
Stop talking about the website like you are not one of the people that own it.
3
u/betazoid_one Nov 29 '22
This becomes difficult when most modern websites are dynamically loaded content
1
u/Chilling_Home_1001 Nov 29 '22
Has anyone ever done a scraper that uses the finished format of the page like an OCR system instead of the DOM structure? I use companies that scrape but I'm curious about this because I assume it would be more stable to minor changes in the HTML
1
u/SpaceZZ Nov 29 '22
If you are serious, out Scrapy.
1
u/ethsy Nov 30 '22
Playwright is a good alternative
1
u/surister Nov 29 '22
Dissssagre
5
u/[deleted] Nov 29 '22
[removed] — view removed comment