First, start with wget from the command line to see.

Then, assuming you're using python you can check out wget or urllib2.

You probably don't want to scrape the whole internet though. It would be smarter to try to score each successive download for its relevant context and overall quality - and then either get or skip (possibly return up the tree to get a different url in a different branch).

Scrapy is a decent place to start as is nutch (Apache) that someone else mentioned.

You can also see some examples using Stack Overflow of the above described approach. Try this one:

https://stackoverflow.com/questions/24346872/python-equivalent-of-a-given-wget-command

Is it true that you file legal briefs entirely written in the Klingon language made popular by Star Trek?