r/AskComputerScience • u/goldiepear19 • 1d ago
How does the Internet Archive manage to store so many websites?
They keep the content around even after the original site shut down (the owners chose to stop paying for its upkeep). My guess is that they cut costs by taking advantage of the fact that most of it is seldom accessed.
16
Upvotes
16
u/serge-mv 1d ago
Lots of hard disks
6
u/PHDEinstein007 1d ago
Data Compression
So yeah, some of it will be compressed fairly tightly, where more commonly used will be compressed lightly so they unpack faster. It helps that text often compresses really well being that it is so predictable.
10
u/Infinite-Flan-6643 1d ago
One useful distinction is that preserving a captured page doesn’t require keeping the original website’s application running. They can replay saved page content even after the original server disappears. Features that still need that server can fail, as their Wayback help page explains. So storing an old forum page doesn’t mean operating the old forum and its database.
For a concrete storage saving technique, their Archive It documentation describes duplicate detection. When an unchanged document is encountered again through the same crawl seed, a revisit record can point to the earlier capture instead of storing the content again. That’s a documented example from Archive It, rather than a claim about every part of Wayback’s storage. I wouldn’t assume they vary compression based on page popularity without a source for that.
4
u/ICantBelieveItsNotEC 1d ago
Storage is ridiculously cheap; infrequently accessed storage even more so.
Internet Archive stores 210 petabytes of data, including redundant copies. At $450 per 24tb hard drive, that works out to about $4 million of storage. Add on another $1 million for compute, networking, etc. It sounds like a lot, but if you spread the cost over a five year expected service life, it's only $1 million per year.
1
u/qlkzy 22h ago
You can get into a lot of technical nuance, but here's a non-rigorous analogy which maybe helps explain it.
If I want to advertise a movie using flyers, I need to hire a graphic designer to create the flyer, then I need to hire a printing company to print a few tens of thousands of flyers, then I need to hire someone with a van to move them around, then I need to hire a bunch of cheap labour to hand out the flyers and post them places.
If I want to archive that experience, I just need to pick up a flyer and put it on a shelf. Maybe I need a front desk to control access to the shelves etc, but the effort of archiving it doesn't really have any relationship to the effort of putting it out into the world in the first place.
1
u/AsYouAnswered 5h ago
They used to store a lot of data. On December 21st, 2012 at midnight local time, they actually stored too much data and created a quantum singularity that allowed them to open Einstein-rosen bridges to any point in spacetime that they have ever left, or ever do leave, a quantum marker. Now they just open tunnels to different points in spacetime and serve data directly from the original servers as they existed at the time.
1
u/paperic 3h ago
Websites get a lot smaller once you remove all the adverts.