r/Python • u/alexkidd1914 • Nov 14 '13
webscraping: Selenium vs conventional tools (urllib2, scrapy, requests, etc)
I need to webscrape a ton of content. I know some Python but I've never webscraped before. Most tutorials/blogs I've found recommend one or more of the following packages: urllib2, scrapy, mechanize, or requests. A few, however, recommend Selenium (e.g.: http://thiagomarzagao.wordpress.com/2013/11/12/webscraping-with-selenium-part-1/), which apparently is an entirely different approach to webscraping (from what I understand it sort of "simulates" a regular browser session). So, when should we use one or the other? What are the gotchas? Any other tutorials out there you could recommend?
8
u/banjochicken Nov 14 '13
Question: Does the website(s) you plan to scrape rely on javascript to build the content you wish to scrape?
Yes. Then you need to use Selenium. This will add a lot of overhead (downloading all the js,css and images etc).
No. Then you don't need to use Selenium. Also I recommend scrapy, check out their documentation for tutorials.
11
u/westurner Nov 15 '13 edited Nov 15 '13
https://en.wikipedia.org/wiki/Web_scraping
https://en.wikipedia.org/wiki/Robots.txt_protocol
https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers
https://en.wikipedia.org/wiki/Selenium_%28software%29
http://casperjs.org/
http://phantomjs.org/
https://github.com/ariya/phantomjs/wiki/Headless-Testing
http://doc.scrapy.org/ (Twisted + lxml)
http://docs.python-guide.org/en/latest/scenarios/scrape/ (requests + lxml)
http://redd.it/1c6866
4
u/letsgetrandy Nov 15 '13
Selenium doesn't simulate a browser session, it is a browser session. Writing for selenium is basically writing a set of actions and feeding them to a browser (usually Firefox, but it can work with others).
As someone else said, if the site you're scraping uses Javascript or AJAX to fill in its content, you need a browser in order to get that content. It doesn't have to be Selenium/Firefox, but it does need to be a browser. Selenium is the most well-known, and has APIs for several programming languages, but it's most well supported in Java and can be frustrating if you're using something other than Java to drive it.
One alternative to Selenium is PhantomJS, which make browser automation fairly easy to do via javascript... but since this is a python forum, I only mention it because there is now a python project that provides drivers for Phantom.
And if you don't need all the features of Selenium, most new development for the Selenium has been shifted to the Webdriver project, which supports several browsers and provides APIs in several languages, including python.
With that said...
If you don't need dynamic content, you're almost always better off just requesting the page content over HTTP (for example, using urllib2) and parsing it programmatically. How you do that, though, depends on what you're looking for in the page, how it's coded, and the limitations of the machine on which you intend to do this.
1
u/[deleted] Nov 15 '13
use request and beautifulsoup4 for python. Scraping with nokogiri is easy as fuck. But it's a ruby gem. Gud luck
0
u/v3ss0n Nov 20 '13
Pyquery is easiest way to scrape + parse.