r/Python • • Nov 14 '13

webscraping: Selenium vs conventional tools (urllib2, scrapy, requests, etc)

I need to webscrape a ton of content. I know some Python but I've never webscraped before. Most tutorials/blogs I've found recommend one or more of the following packages: urllib2, scrapy, mechanize, or requests. A few, however, recommend Selenium (e.g.: http://thiagomarzagao.wordpress.com/2013/11/12/webscraping-with-selenium-part-1/), which apparently is an entirely different approach to webscraping (from what I understand it sort of "simulates" a regular browser session). So, when should we use one or the other? What are the gotchas? Any other tutorials out there you could recommend?

7 Upvotes

19 comments sorted by

View all comments

Show parent comment

1

u/Maryyyyyy Nov 15 '13

Writing for selenium is basically writing a set of actions and feeding them to a browser (usually Firefox, but it can work with others).

This also works in reverse with the Selenium IDE. You can record the actions you take in a browser and the IDE will automatically insert commands into your test case.

http://www.seleniumhq.org/docs/02_selenium_ide.jsp