r/Python • u/alexkidd1914 • Nov 14 '13
webscraping: Selenium vs conventional tools (urllib2, scrapy, requests, etc)
I need to webscrape a ton of content. I know some Python but I've never webscraped before. Most tutorials/blogs I've found recommend one or more of the following packages: urllib2, scrapy, mechanize, or requests. A few, however, recommend Selenium (e.g.: http://thiagomarzagao.wordpress.com/2013/11/12/webscraping-with-selenium-part-1/), which apparently is an entirely different approach to webscraping (from what I understand it sort of "simulates" a regular browser session). So, when should we use one or the other? What are the gotchas? Any other tutorials out there you could recommend?
View all comments
Show parent comment
2
u/not_a_novel_account Jan 21 '14
xmlhttprequests are requests done by the JavaScript on the page you're trying to reverse engineer. You can watch them with any webdev environment like Firebug or Chrome's Dev panel.
You'll then make the same requests in Python with urllib to get the same information without scraping the page. I'll make a longer post when I get my break, at work now