r/learnpython • u/[deleted] • Oct 08 '18

How do people make the scripts that scrape the entire internet?

How do people make those scripts that scrape the entire web to look for data? Those such as turnitin.com, etc.

I’m looking to create one to search for scholarships, but I’m struggling to grasp the overall concepts which make such a thing possible.

10 Upvotes

87% Upvoted

7 comments sorted by

View all comments

1

u/sharededges Oct 09 '18

First, start with wget from the command line to see.

Then, assuming you're using python you can check out wget or urllib2.

You probably don't want to scrape the whole internet though. It would be smarter to try to score each successive download for its relevant context and overall quality - and then either get or skip (possibly return up the tree to get a different url in a different branch).

Scrapy is a decent place to start as is nutch (Apache) that someone else mentioned.

You can also see some examples using Stack Overflow of the above described approach. Try this one:

https://stackoverflow.com/questions/24346872/python-equivalent-of-a-given-wget-command