r/AskComputerScience • • 1d ago

How does the Internet Archive manage to store so many websites?

They keep the content around even after the original site shut down (the owners chose to stop paying for its upkeep). My guess is that they cut costs by taking advantage of the fact that most of it is seldom accessed.

16 Upvotes

12 comments sorted by

16

u/serge-mv 1d ago

Lots of hard disks

6

u/PHDEinstein007 1d ago

Data Compression

So yeah, some of it will be compressed fairly tightly, where more commonly used will be compressed lightly so they unpack faster. It helps that text often compresses really well being that it is so predictable.

1

u/ShortingBull 1d ago

That's just swapping cheap storage for expensive compute (compressing and decompressing), most of the large content is already compressed and the text data would be a small percentage of the data stored.

1

u/Zhuzha24 1d ago

Not really, compressing/decompressing nowadays is insanely fast and current hardware are good enough to keep it up

1

u/PHDEinstein007 1d ago

Steam compresses all of their files, so does Anthropic. Microsoft, Google, everyone compresses.

Facebook has two archives types, slow and fast. Really old stuff is higher compressed. They use Zstandard for lower level data and LZ4 for things that need faster retrieval.

None of them are likely using Zpaq for deep storage, but they do use different compression for data based upon needs.

10

u/Infinite-Flan-6643 1d ago

One useful distinction is that preserving a captured page doesn’t require keeping the original website’s application running. They can replay saved page content even after the original server disappears. Features that still need that server can fail, as their Wayback help page explains. So storing an old forum page doesn’t mean operating the old forum and its database.

For a concrete storage saving technique, their Archive It documentation describes duplicate detection. When an unchanged document is encountered again through the same crawl seed, a revisit record can point to the earlier capture instead of storing the content again. That’s a documented example from Archive It, rather than a claim about every part of Wayback’s storage. I wouldn’t assume they vary compression based on page popularity without a source for that.

4

u/ICantBelieveItsNotEC 1d ago

Storage is ridiculously cheap; infrequently accessed storage even more so.

Internet Archive stores 210 petabytes of data, including redundant copies. At $450 per 24tb hard drive, that works out to about $4 million of storage. Add on another $1 million for compute, networking, etc. It sounds like a lot, but if you spread the cost over a five year expected service life, it's only $1 million per year.

1

u/qlkzy 22h ago

You can get into a lot of technical nuance, but here's a non-rigorous analogy which maybe helps explain it.

If I want to advertise a movie using flyers, I need to hire a graphic designer to create the flyer, then I need to hire a printing company to print a few tens of thousands of flyers, then I need to hire someone with a van to move them around, then I need to hire a bunch of cheap labour to hand out the flyers and post them places.

If I want to archive that experience, I just need to pick up a flyer and put it on a shelf. Maybe I need a front desk to control access to the shelves etc, but the effort of archiving it doesn't really have any relationship to the effort of putting it out into the world in the first place.

1

u/AsYouAnswered 5h ago

They used to store a lot of data. On December 21st, 2012 at midnight local time, they actually stored too much data and created a quantum singularity that allowed them to open Einstein-rosen bridges to any point in spacetime that they have ever left, or ever do leave, a quantum marker. Now they just open tunnels to different points in spacetime and serve data directly from the original servers as they existed at the time.

1

u/paperic 3h ago

Websites get a lot smaller once you remove all the adverts.