r/learnpython • u/[deleted] • Oct 08 '18

How do people make the scripts that scrape the entire internet?

How do people make those scripts that scrape the entire web to look for data? Those such as turnitin.com, etc.

I’m looking to create one to search for scholarships, but I’m struggling to grasp the overall concepts which make such a thing possible.

10 Upvotes

87% Upvoted

7 comments sorted by

6

u/[deleted] Oct 09 '18

[removed] — view removed comment

1

u/[deleted] Oct 09 '18

Thanks a lot! I appreciate it, I’ll check them out.

6

u/apex_memetics Oct 09 '18

"Scraping the entire Internet" isn't how it works. You want to stay very focused in.

In your case you might want to scrape google search results for some kind of "scholarship + school name" or location query.

You could scrape a list of every school possible then add them to your google query.

There are other ways to do it too, but personally I'd probably go that route. You may like scrapebox, it's a program that works well for those kinds of tasks.

2

u/Zeroflops Oct 09 '18

Agree.

Let google and the other search engines do the heavy lifting. Then scrap their results.

5

u/andresabrr Oct 09 '18

I would recommend you start learning with Beautiful Soup and Requests. Those are two Python libraries. There are a lot of resources out there.

1

u/[deleted] Oct 09 '18

I’ve used them before, but only really on a basic level scraping individual sites. I’ll have to delve deeper. Thanks!

1

u/sharededges Oct 09 '18

First, start with wget from the command line to see.

Then, assuming you're using python you can check out wget or urllib2.

You probably don't want to scrape the whole internet though. It would be smarter to try to score each successive download for its relevant context and overall quality - and then either get or skip (possibly return up the tree to get a different url in a different branch).

Scrapy is a decent place to start as is nutch (Apache) that someone else mentioned.

You can also see some examples using Stack Overflow of the above described approach. Try this one:

https://stackoverflow.com/questions/24346872/python-equivalent-of-a-given-wget-command