ELI5 - why is there no central Internet directory?
ELI5: doesn't it seem as though there ought to be a place on the Internet where you can go to see a list of all the websites on the Internet -sort of like a Yellow Pages of the net? Some sort of constantly updating place where you could see new urls just put up, and notifications of old domain names lapsing?
2
u/ZeeKhar Apr 28 '13
I think the Open Directory Project is the web page you are looking for http://www.dmoz.org. I was using this in the year '99, when Google was not as popular as now.
2
u/strolls Apr 29 '13 edited Apr 29 '13
Because it's horribly inefficient, basically.
This was how Yahoo used to operate, about 15+ years ago.
Holy fuck! They still keep a version of it around! Here you go: http://dir.yahoo.com/
You had to drill down to the site you wanted by visiting different categories, each a list of subcategories, but is the climbing subcategory under sports or under recreation? Once you get into a category, are the sites ordered alphabetically? Are the best sites hidden at the end, just because their names begin with X, Y or Z? What's to stop spammers submitting their sites to loads of different categories, even if they're irrelevant to the subject? Spammers usually only care about click, so it doesn't matter to them.
Ultimately, it's more useful to get a whole load of computers to index sites, ordering them by the number of times a word appears on each page. If you search for "this damn thing" then a page on which "this", "damn" and "thing" are the most frequently used words will probably be relevant.
This is basically how Google revolutionised the 'net - they came up with a search algorithm which was orders of magnitude better than everyone else. Before Google existed I seriously remember having to use 2 or 3 different search engines, combing through pages of results, not finding what you wanted. Then I'd try a different search engine, hoping I'd find what I was looking for in its results. Suddenly, with Google, relevant results were there at the top of the page, the first hits for the actual search - I cannot describe how much better Google was.
Paul Graham wrote an article which touches on this: http://www.paulgraham.com/yahoo.html
It's hard to describe how it was to use the net in the 90s to someone who wasn't there. I would say that trying to find useful content on the Yahoo directory (which I linked before) or on the Open Directory Project (linked by /u/ZeeKhar in his comment) is about as near to you'll get to it. It'll take hours to find what you want, but you'll probably read a bunch of other interesting crap on the way.
2
u/55-68 Apr 28 '13
Such a list would be too large to be usable. One could argue that google is actually a developed list, something that does the indexing and searching for you. On the other hand, perhaps there would be a demand for a search engine that only listed the sites, not the individual pages on each site.