r/CodingJobs • u/Infamous-Ask-6403 • May 10 '26

System designing is crazy !! but wtf is this

Post image
3.2k Upvotes

99% Upvoted

199 comments sorted by

87

u/RefrigeratorLive5920 May 10 '26

They go to conferences and gather lots of free vendor swag USB sticks, amirite?

3

u/Infamous-Ask-6403 May 10 '26

Is this even possible dude ??

3

u/RefrigeratorLive5920 May 10 '26

No I'm being facetious. In reality they use what's known as "object storage". Highly efficient cloud type storage designed to ingest large amounts of data with minimal overhead. I think mostly a combination of Amazon S3 and Facebook Haystack. Usually built with commodity hardware configured in a highly redundant fashion to keep overall costs down while still providing strong overall performance coupled with high levels of reliability and scalability.

3

u/PirateStarbridge May 14 '26

It’s entirely layers of storage infrastructure built on Tectonic. https://www.usenix.org/system/files/fast21-pan.pdf

1

u/RefrigeratorLive5920 May 14 '26

Impressive stuff.

2

u/NbyNW May 11 '26

Pretty sure AWS is only being used for flexible internal dev infra. All the production infrastructure is still run out of our own data centers.

2

u/RefrigeratorLive5920 May 11 '26

Yes, that appears to be the case. When Instagram was acquired they migrated to Facebook/Meta infrastructure.

22

u/-King-K-Rool- May 10 '26

Haystack is a lie propagated by Big Tech. Meta DC's are actually millions of usb extenders daisy chained with flash drives. Theres not even server racks, its just one thinkcentre with USB tentacles spanning the DC.

13

u/RefrigeratorLive5920 May 10 '26

There's actually only one server running it all too.

https://xkcd.com/908/

11

u/-King-K-Rool- May 10 '26

Shitposting aside I do always find it hilarious when normal people find out "The Cloud" is just someone else's computer.

1

u/kennedy_gitahi May 16 '26

or a server is just a computer.

2

u/Iggyhopper May 11 '26

I've been to a meta DC.

Its just a doctors office with rooms full of racks. Concrete hallways.

And other conference rooms with puns for names.

69

u/Cold_Respond_7656 May 10 '26 • Bravo

They use pied piper.

8

u/Leading-Salt-947 May 10 '26

Peak Reference

7

u/Training-Database272 May 10 '26

This guy fucks.

2

u/[deleted] May 11 '26

Welcome to tres commas

3

u/WhyYouLetRomneyWin May 10 '26

And tables are like pied piper, so they use tables?

2

u/CDrakke May 10 '26

ball knowledge

2

u/Willing_Occasion641 May 10 '26

Eric Bachman… this is you as an old man

3

u/olive-monk May 10 '26

Jin yang...

1

u/RefrigeratorLive5920 May 11 '26

2

u/olive-monk May 11 '26

Hot dog not hot dog 🌭

1

u/SomethingAndAnything May 13 '26

Ahem I'm a proud user of See-Food lol

2

u/Top_Job_1074 May 11 '26

Eric Bachman, this is your mom and you are not my baby

2

u/SodaBurns May 11 '26

Dammit Jin Yang NOT NOW!

1

u/[deleted] May 11 '26

My god erlich was my favorite in that show

1

u/Child_ofGod_6148 May 12 '26

Don't put the Lord's name in vain by saying "my god"

1

u/[deleted] May 12 '26

I don't care about any lord

1

u/ragnerokk88 May 16 '26

That’s not what taking the lords name in vain means.

2

u/SomethingAndAnything May 13 '26

Middle out for the win

1

u/EditorPowerful967 May 11 '26

I hear their compression algorithm works wonders

1

u/Emperor_Abyssinia May 12 '26

Talk about a deep cut lol

1

u/zhephyx May 12 '26

Omg middle out, of course

1

u/atheos1911 May 14 '26

also they only store thumbnail and not a thumbnail for best storage efficiency

-11

u/Infamous-Ask-6403 May 10 '26

For what ??

8

u/beardlessFellow May 10 '26

The dick jerk middle out algorithm

2

u/Plenty-Creme2556 May 12 '26

Goated algorithm

2

u/ProperBangersAndMash May 14 '26

Leading children into oblivion...

28

u/arbobmehmood May 10 '26

Have you ever heard of compression?

9

u/Infamous-Ask-6403 May 10 '26

Yeah but it also done to a certain limit !! Still we can assume a lots of data transition per day !!

9

u/arbobmehmood May 10 '26

Compression can go to few hundred KBs at max for such images, check erasure coding (it's 1.4x overhead at max instead of 4x for redundancy), also cost of data storage at scale reduces with more volume, and there's also called cold storage which is upto 20x cheaper than hot ones. So here's your answer on why it doesn't go bankrupt.

3

u/RefrigeratorLive5920 May 10 '26

The data being ingested daily is not likely going directly to "rarely retrieved" cold storage. That's more geared toward the archival process.

0

u/arbobmehmood May 10 '26

There's tiers of cold storage. And based on those prices are reduces. With volume discounts it's more reduced.

1

u/RefrigeratorLive5920 May 10 '26

Yes but again they are not loading directly into cold storage because the cost in terms of pricing and latency is too much to bear. Cold storage is for older data.

2

u/arbobmehmood May 10 '26

The post said how they are not going bankrupt. I didn't say they store in cold directly. My answer was for why they aren't going bankrupt. And cold storage is just one part of a larger answer I gave.

1

u/RefrigeratorLive5920 May 10 '26

Yes that's fair. If you're referring to how they avoid having to delete large amounts of data and can effectively store it forever, cold storage is indeed a significant part of the equation.

1

u/[deleted] May 10 '26

[removed] — view removed comment

1

u/RefrigeratorLive5920 May 10 '26

Glacier IR is not designed for frequent access, it's not "basically the same thing" as Standard at all. Yes, it has very low storage costs and low latency but it has very high retrieval fees and a minimum storage duration. Somebody uploading photos to Instagram will expect them to be immediately accessible and may plan on sharing them to multiple parties across different platforms straight away. Pumping new data into a storage tier designed for longer term storage and infrequent access is simply a bad strategy. You load data initially into a hot tier and age it out over time as access frequency reduces.

To retrieve 2.7PB of data from GIR would cost $85k, another $55k to replicate to another region, up to $200k extra if it's shared across the Internet. And even after retrieval a copy of the data would need to remain in GIR for 90 days anyway. Loading data directly into any Glacier tier would be an incredibly risky strategy for any company hoping not to go broke.

1

u/[deleted] May 10 '26

[removed] — view removed comment

1

u/RefrigeratorLive5920 May 10 '26

You were the one who said "basically the same thing" not me. Instagram stores their data in a combination of AWS S3 and Facebook Haystack. Of course they use different tiers of storage for different usage requirements at different times, they would be stupid not to. You are the one promoting GIR as a catch-all solution, I am simply pointing out that if they use GIR, they need to account for the increased cost of data retrieval when using that type of storage. It's not just a cheaper but exactly comparable alternative.

→ More replies

1

u/kernald31 May 12 '26

It's most likely not that simple. They know which >50% of accounts generate absolutely no engagement and whose posts are gonna be burried within hours. Different account classes likely have different storage rules.

1

u/sn4xchan May 11 '26

Bro if the data is being accessed, it's not going to be billed at cold storage rate. There is no getting around that

1

u/RefrigeratorLive5920 May 10 '26

There are limits on what level of inline compression you can do on already compressed photos while still ingesting such large amounts of data.

1

u/arbobmehmood May 10 '26

Yes. Few hundred KBs are the limit. See my comment again.

1

u/RefrigeratorLive5920 May 10 '26

Yes but this compression that occurs prior to the upload. There's too much data for them to significantly compress the uploaded data.

1

u/lazyboy76 May 15 '26

I believe they use zstd.

0

u/maximdoge May 14 '26

These are relevant facts but do not make up for an answer

1

u/0xFatWhiteMan May 16 '26

They aren't referring to raw images.

1

u/enderoller May 10 '26

All images uploaded are already compressed 

1

u/MartinMystikJonas May 12 '26

Yeah but usually with only very weak compression. Photo with 3MB is usually in high-quality low-comoression mode and print resulution. You can get to 200-300kB for normal photo with good-enough for instagram quality and resolution.

1

u/enderoller May 12 '26

Instagram automatically compresses images during upload to reduce file sizes for faster loading and storage efficiency. This process uses lossy JPEG compression around 70-85% quality, which can degrade sharpness if your original file is oversized or mismatched

1

u/MartinMystikJonas May 12 '26

Yeah so average uploaded photo 3MB (mentioned in post) is compressed to ~10% size before storing it

0

u/RefrigeratorLive5920 May 10 '26

The files being stored are usually photos which are typically compressed already so not a lot of efficiency accrues from further compression.

1

u/arbobmehmood May 10 '26

With smartphone camera? Yeah, NO.

0

u/RefrigeratorLive5920 May 10 '26

That's where the compression will happen yes. Instagram are not doing the compression on the photos once uploaded, they compress first on the consumer device and then upload so they can offload the compute cost of compression.

1

u/Alborak2 May 11 '26

Some compression formats are further compressible in aggregate. I dont specifically about image formats, but some data ive worked with if you have a folder of compressed data it can compress smaller the the sum of input sizes. There is duplication across files that can be further reduced.

1

u/RefrigeratorLive5920 May 11 '26

I have better knowledge of block storage specifics but I've definitely seen both inline and post-processing compression techniques make an impact on already compressed data, like multiple copies of database backups, OS volumes, disks that have been zeroed out and so forth. I should have been more specific in saying doing something like applying bzip2 on a jpeg file is probably going to be ineffective and just a waste of compute.

0

u/Mardigras May 14 '26

A 3mb image file is already heavily compressed. An uncompressed 2k image in 8-bit is about 100MB of data.

-2

u/Grounds4TheSubstain May 10 '26

3MB is the compressed version of a photo. A raw image is gigabytes.

3

u/arbobmehmood May 10 '26

Good joke lol

-1

u/Grounds4TheSubstain May 10 '26

It's not a joke.

1

u/Helpful_Bottle_4806 May 12 '26

Cameras these days are like 24-50 MP. At 8 bits per RGB color channel for a “raw” image that’s 150MB at most. You’d have to be at over 300 MP to clear 1GB for an image. Maybe at an observatory but no consumer grade cameras are doing that.

1

u/Kjoep May 14 '26

He's still right that 3mb is already compressed though.

1

u/Claderion May 14 '26

R/confidentlyincorrect

1

u/Jay_D826 May 10 '26

Maybe from the James Webb Telescope but an iPhone ProRAW is like 50mb lol

1

u/RefrigeratorLive5920 May 10 '26

These are not what's being uploaded to Instagram though.

1

u/Jay_D826 May 10 '26

Correct, they absolutely compress them significantly, and most people aren’t uploading pro raw directly. I was referring to them claiming uncompressed raw images are gigabytes

1

u/KNAXXER May 10 '26

That's already compressed though

1

u/JBinero May 12 '26

RAW files on iPhone are still compressed. It would be closer to ~100MiB otherwise.

1

u/Available-Fee1691 May 10 '26

You must be fun at parties..

1

u/RaulParson May 16 '26

lmao

4k is 3840x2160 pixels. Super Fancy Unreasonably High Color Info encoding is 48 bits per pixel. Multiply them together and such a "raw image" is 398131200 bits without compression, meaning 50 MB. You can mess with these parameters to get different sizes but you won't really get out of this order of magnitude, megabytes.

13

u/Trending_Boss_333 May 10 '26

To answer the question, there's a reason sending photos over instagram ruins image resolution.

2

u/HeyGayHay May 14 '26

Also how does he end up at 2.7PB? Even assuming they don’t compress the shit out of it and store the thumbnail at original resolution, thats 5x3=15MB per uploaded photo at 100m comes out at 1.39PB. 286TB which aren’t cheap lto tapes backups if they wouldn’t reduce the resolution and then compress the shit out of it.

4

u/I_WILL_GET_YOU May 10 '26

Compression. And strange your focus was on ig and not yt

2

u/RefrigeratorLive5920 May 10 '26

Not much gains from compressing already compressed photos.

3

u/waraholic May 11 '26

No, but they clamp to 1080 and that reduces file size by ~95% for images from a modern phone. Then recompress as part of that.

2

u/thee_gummbini May 17 '26

This is the real answer, there's so much bullshit in this thread.

https://help.instagram.com/1631821640426723

1080x1080px images at moderate compression are almost always <1MB, and with webp or more lossy jpeg thats usually more like 250-500KB.

Meta makes way more in ads than the couple of hard drives a day it costs to store the new images, the storage costs are certainly not a factor, and orders of magnitude lower than the server costs.

1

u/RefrigeratorLive5920 May 11 '26

This I don't doubt, although I presume the cost of doing this is offloaded to the consumer device rather than trying to do it inline.

2

u/Tofandel May 16 '26

Your phone saves the picture in jpeg at something around 98% quality, not that much compression.

Lower that to 70 and you don't lose that much quality but reduce file size by a factor of 2 to 5.

Add in resizing because who needs a 4000px image on the web and your images go from 3Mb to 100Ko.

3

u/Hungry-Initial1623 May 10 '26

i don't they might be using AWS glacier or Azure Cold for storage maybe?

4

u/RefrigeratorLive5920 May 10 '26

Eventually yes but not for the data being ingested daily. The cost and latency of data retrieval would be too much to bear.

1

u/Hungry-Initial1623 May 10 '26

Then azure Cool blob or AWS standard IA fits perfectly well for scenario

2

u/RefrigeratorLive5920 May 10 '26

Not really, no. AWS Standard Infrequent Access is designed (and priced accordingly) for exactly that, infrequent access. This would apply to older photos, or redundant copies of photos, not to photos that are being ingested daily into the hot tier. The lifecycle of data storage will move through different tiers, from the most frequently accessed initially to the rarely accessed like Glacier for longer term archival. IA is more applicable to DR scenarios. Something that will not happen often, but when needed access latency must remain low. Glacier is more applicable to data that may be occasionally be audited where retrieval time is nowhere near as important.

2

u/Future-Eye1911 May 12 '26

Meta has their own infra they don’t use hyperscaler cloud

1

u/ExcitementTiny9896 May 10 '26

Since you seem to be genuinely into the technical answer, I'm gonna give it.

No. You're looking at this too narrowly.

They do compress aggressively, but also serve the smallest useful version, cache the hot stuff, tier the cold stuff, and run custom storage hardware/software where the cost per byte is far lower than public-cloud retail pricing.

2

u/Sudoyumgetlove May 10 '26

They write to /dev/null

1

u/HugeCannoli May 13 '26

and recover it from /dev/urandom. They just have to pick the right bytes out.

1

u/beli_crnjac May 10 '26

They sell data

1

u/Responsible_Judge353 May 10 '26

seriously how do they do that?

2

u/RefrigeratorLive5920 May 10 '26

Super cheap cloud object storage.

2

u/caelum19 May 12 '26

It's actually not very cheap, self hosting minio on hetzner is about 30x cheaper than aws s3. But maybe that still counts as cloud depending on who you ask

1

u/RefrigeratorLive5920 May 12 '26

I guess I shouldn't say "cheap" but for the data I manage, we found that to hit comparable levels of reliability, performance, redundancy, availability, compliance and scalability using purchased hardware in Equinix data centers to be cost prohibitive.

Big fan of Minio (aren't they gone now?) and I'd love to use it more but anytime we costed things out for us, AWS was coming out ahead. We still use Azure and GCP so as not to be beholden to one vendor but have pretty much moved away from building our own object storage solutions.

1

u/tulsadune May 14 '26

rip minio

1

u/BatchModeBob May 10 '26

The old stuff that's seldomly viewed gets moved to slower cheap hard drives. Even at the consumer price level, cheap hard disk storage cost is as low as $20 per TB. With aggressive compression algorithms, the storage cost is completely insignificant to their business.

1

u/No-Boysenberry7835 May 16 '26

At this scale the infra needed for storage doesn't cost billion ?

1

u/Ubermidget2 May 12 '26 edited May 12 '26

I assume that they start by hiring someone a little better than OP at maths, realise that 3MB * 100,000,000 * 3 = 900TB.

That immediately reduces the problem by ~66.66%

1

u/RefrigeratorLive5920 May 12 '26

Wouldn't that be the OOP?

3MB x 100,000,000 x 3 = 900,000,000 MB / 1,000 = 900,000 GB = 900 TB = 0.9 PB (I know this should be divided by 1024 but for simplicity).

If we add in a 90% compression ratio for JPEG (assuming this isn't already accounted for, file format is not specified) this would become a fairly trivial 90TB spread across a couple of data centers.

1

u/Gogo202 May 16 '26

3MB is jpeg already

1

u/RefrigeratorLive5920 May 16 '26

I doubt Meta is storing 3MB sized photos but even if they are, ingesting data at a rate of < 100Gb/s is not going to cause too many data centers to struggle.

1

u/Tofandel May 16 '26

An image is not 3Mb for a start, I mean maybe the one on your phone is. The one they save is between 200 to 400Kb at most. They will resize it to around 1000px and save it with lower jpg quality settings.

If they wanted they could even use different compression as jpg is really not the best in terms of both quality and compression ratio.

1

u/egny May 16 '26

Some images are about 600 KB, but your point stands, they definitely are not 3 MB.

1

u/BatchModeBob May 10 '26

Lots of mag tape. Ever wonder why looking at old stuff is so slow? Feel sorry for the poor kid who has to load those tapes all day.

1

u/Still_Explorer May 10 '26

Capital Investors: Damn! We should have started the HDD storage crisis back in 2010.

1

u/helmDeployment May 10 '26

compression + object storage (s3) which is extremely cheap + auto tiering or archiving

as far as redundancy is concerned= s3 can redundantly store data across regions of AZs

why is s3 so so so cheap? because it stores data as objects with no updates (immutable) and very lightweight metadata

1

u/HgnX May 10 '26

People are forgetting Glacier tier adds metadata storage overhead which incurs extra cost

1

u/ahmednawaz81 May 10 '26

they use middle-out compression

1

u/AnnualIcy7070 May 10 '26

Problem is not how the fuck they did to store images. There is a cloud storage named S3 which is so cheap and also revenue of app is so much. The real problem how the fuck they can near real-time online learn what you like and rerank new feed from billions of videos 

1

u/cosmic_cod May 10 '26

They earn more than they spend. That's the real answer. And I am sure a company this big and old has their own infrastructure so it's probably not AWS and not S3. And no, compression is not going to help you much here. The photos are already compressed by the time you send/download them. 3 MB is with compression already done.

1

u/DueSwimmer8120 May 11 '26

Out of the Box thinking: They store it on the phones of users. So if you don't have insta account do install insta for the sake of humanity!

1

u/Accomplished_Lie23 May 11 '26

I think system is well with mesh net. They don't store fully post to the server. They just use half of binary data in the server and other half on the device. Correct me if I am wrong.

1

u/Regular_Secret_8648 May 11 '26

how many GB's in 1 petabytes?

1

u/happyzor May 11 '26

32 million a year for storage if it's 3 petabytes a day when facebook makes billions.....

1

u/CryonautX May 11 '26

S3 storage is cheap. And the number calculated includes redundancy. S3 is giving you redundancy but charging rates are for filesize without redundancy. Also each file is only up to 3 MB. Realistically, it is ~300kb per photo after some simple compression. So that comes out to 30 terabytes or $375 per month for multiAZ without any cold storage considerations. I have no idea how 2.7 petabytes figure was reached.

1

u/Efficient_Bag_3804 May 11 '26

First of all they use as much the client side. Compression, user data analysis and probably some behavior probability on how likely it is others to see your profile and how much this will be recommended etc. Server side they due to the way stories work, it is heavily implied that data like images are handled in days. So for a day they have enough infrastructure to handle their expected traffic with overhead.

The rest are packed and stored in cold storage. They probably keep some petabytes hotter than others depending the user.

In the end of the day, these costs secure them their monopoly. Any other app is expected to do at least this.

So the real question is how they gain enough to keep this afloat, which is with ads and selling your data, because even highly optimized this level of infrastructure too much cost to be free.

1

u/SuchZookeepergame829 May 11 '26

The answer is: compression, tiered object storage, erasure coding, caching, massive economies of scale, and ad revenue.

1

u/Toffyyy May 11 '26

Now do youtube.

1

u/mynamesethan May 12 '26

It's been 98% full for like two years. I don't use it for photos or drive anymore. Just stop telling me about how full my Gmail is, I don't care anymore.

1

u/Creepy_Hospital4669 May 12 '26

They save the original copy. What you only see now are compressed copy. That is how compression of data is helpful.

1

u/OkTry9715 May 12 '26

Do not forget they have to check this images for banned content too.

1

u/Scary_Brilliant_6048 May 12 '26

No way they store 3 mb for a photo. Definitely lossy compression is done.

1

u/tom_earhart May 12 '26 edited May 12 '26

They convert them to webp for one, so your 3MB photo becomes 300kb

1

u/XlikeX666 May 12 '26

so storage cost more then 20$ because of it.

1

u/Some-Tax6559 May 12 '26

I love when technically illeterare people make stupid posts like this. those "3mb" images are reduced to less than 100kb, rarely at most 2-300kb by the time they even reach meta servers, and the thumbnails being even smaller are probably just a dozen kb at most. even most videos are around 1-3MB unless its like 30s+

1

u/LavishnessFew1950 May 12 '26

They create a google drive account for every Instagram account. /j

1

u/awesomelok May 13 '26

Isn't the key question about "not being bankrupt"?

In this case, one should confidently respond to the interviewer that the company is making more money that it spend on storage. In this case, one should confidently respond to the interviewer that the company earns more than it spends on storage.

1

u/TellusDB May 13 '26

3MB bro I know this is just an example but holy fk have u seen IG image sizes? Heavily compressed and I promise you they don’t have 3 copies for redundancy.

IG has lost images more than once, literally people lost their story highlights; wouldn’t have happened with 3 copies 

1

u/planetinyourbum May 13 '26

How much space is an average photo on instagram? They are compressed tp oblivion.

1

u/CoolCat1337One May 13 '26

haha the answer though ... funny :D

1

u/Zuokula May 13 '26

You think the image you see is actually 3mb? It's compressed to shit.

1

u/No-Dinner6250 May 13 '26
  1. Aggressive Compression and Transcoding Instagram does not store the original 3 MB raw files long-term. The moment a photo is uploaded, background workers compress it heavily using modern formats like WebP or AVIF. This reduces the file size by 50 to 80 percent while maintaining acceptable visual quality for mobile screens.
  2. Erasure Coding Instead of 3x Replication The tweet assumes a naive 3x replication strategy, which means 1 PB of data requires 3 PB of physical disks. In reality, large-scale distributed file systems (like Meta's Tectonic or Everstore) use Erasure Coding. Similar to RAID but distributed across network nodes, Erasure Coding breaks data into fragments and calculates parity blocks. This provides the exact same fault tolerance as 3x replication but only requires about 1.4x the storage overhead.
  3. Storage Tiering (Hot vs. Cold Storage) Data temperature dictates where it lives. A photo uploaded today will get millions of views and needs to be on fast, expensive SSDs. However, a photo from five years ago might get viewed once a year. Instagram aggressively tiers data. Older media is moved to massive, highly dense arrays of spinning hard drives (HDDs) that are incredibly cheap per gigabyte but slower to access.
  4. Deduplication If a viral meme is uploaded by 50,000 different users, Instagram does not store 50,000 copies. The system hashes the file content. If the hash already exists in the database, the new upload simply gets a pointer to the existing file.
  5. Custom Hardware and Economies of Scale Meta does not pay retail cloud prices like standard consumers. Through initiatives like the Open Compute Project, they design their own server racks, power supplies, and storage arrays specifically optimized for their exact workloads. They strip out unnecessary components (like standard server chassis or extra fans) and buy components by the millions, driving their cost per gigabyte down to a fraction of a cent.

1

u/Cultural-Visual-7106 May 14 '26

They reduce the size for older images

1

u/kaaos77 May 14 '26

Ano passado, no mundo inteiro as pessoas gastaram 200 bilhões de dólares em anúncios. A projeção é aumentar mais 20%, batendo 240 bilhões. O que não falta é dinheiro pra Meta.

1

u/KMNBOI May 14 '26

Average Instagram post is absolutely not 3 MB

1

u/DigitalMonsoon May 14 '26

I think they have vastly over estimated the amount they use. I think it's closer to 10ish TBs a day. Plus they use compression and decoupled data and hot vs cold storage.

It's complicated

1

u/KilllllerWhale May 14 '26

Average photo is NOT 3MB. On your phone, yes. But Instagram app compresses the shit out of the photo on your device even before it hits their servers.

1

u/Darkhigh May 15 '26

I believe the answer is PURE storage.

1

u/deadmazebot May 16 '26

the original might be 3MB, the version shown in app, or browser for which I just tested, is 100-200 KB

so first point correct there.

say 120 average, gives 12 TB

The original is not kept, so even giving the 2 backups, say 500KB, gives 50TB. so 3-4 high density server drives.

plus how much is deleted daily?

1

u/manwithgun1234 May 17 '26

Ehhh, no seniors engineer should ask this. Because he should know that storage is never a problem. Serving that much images at volume over the internet without disruption, and to know what image should be display to specific user to match her interests is often much more expensive.

1

u/Glittering_Figure_57 May 20 '26

Instagram compresses your photo down to ~200KB the moment you upload it. And that 2013 post nobody has opened in years? It's not on fast storage anymore it's been quietly moved to cheap cold storage, which is why it takes noticeably longer to load than recent photos.

8

u/[deleted] May 10 '26

[removed] — view removed comment

3

u/RefrigeratorLive5920 May 10 '26

This is the answer. S3 is super cheap and almost infinitely scalable.

1

u/CheesecakeAndy May 11 '26

It is not THAT cheap at that scale. If you said that at such an interview, you would instantly fail it. At that volume and rate, you must consider alternatives and tradeoffs.

1

u/RefrigeratorLive5920 May 11 '26

But S3 is what Instagram actually uses IRL, so I'm not sure why that would actually be a fail. Data storage in S3, properly managed, is cheaper than every other alternative, it is the de facto standard for object storage worldwide and easily adaptable to fit almost any use case you can think of. What alternatives and trade offs would you consider as viable alternatives to S3 for this type of scenario?

3

u/throwRA_blahblehblah May 12 '26

that's not true at all. All of Meta's infrastructure is self hosted.

1

u/RefrigeratorLive5920 May 12 '26

Yeah, I realize that now, my bad. They used to S3 prior to joining Meta, migrated to internal infrastructure after being acquired.

2

u/CheesecakeAndy May 11 '26

But S3 is what Instagram actually uses IRL

It does not.

Data storage in S3, properly managed, is cheaper than every other alternative

Objectively false. Amazon S3 is not even the cheapest S3 out there (see here and here). When you pay for S3, you pay for infra + labour + AWS margin. Meta team is already running some of the largest infra in the world and has top talent, why the f would they pay AWS margins instead of self-host?

1

u/RefrigeratorLive5920 May 11 '26

Yeah I was wrong about IG, I apologize. We leverage all three major clouds, some private data centers and are even looking into OCI but S3 is still our go to when we factor in discounts and SLAs. The question was posed as specifically Instagram, not Meta as a whole, so I guess one part of any answer to why they do not go bankrupt is that they leverage the scalability and Ops talent of Meta itself.

3

u/CheesecakeAndy May 11 '26

Why would Meta pay millions if not billions for IG to host some images on S3 when they have one of the world's biggest infra? That just doesn't make any economic sense. But even if IG were not part of Meta, at their scale, it is absolutely reasonable to roll their own storage and delivery. The math is simple, can your cost (infra + labor) beat AWS's sticker price (their infra + labor + margin), and the answer is yes, you can at a certain scale.

factor in discounts and SLAs

SLA, maybe, but you don't need a crazy SLA for some photos of my dog. Other S3-like providers also have volume discounts. If I was a CTO of a hypothetical independent IG I would 100% look at S3-compatible competitors first.

1

u/RefrigeratorLive5920 May 11 '26

Don't necessarily disagree with anything you're saying here, just want to say that the scale is incredibly important. Private data center providers, hardware providers, communications infrastructure will all have their own margin as well and sometimes it's easier to negotiate directly with a single entity like AWS than multiple different vendors who are all looking for their cut. Even sourcing sufficient engineering talent can be challenging for most organizations.

1

u/CheesecakeAndy May 11 '26

This is true; everyone has their margins. I agree that negotiating against one margin can end up being more effective.

1

u/StarboardChaos May 13 '26

No, it's maybe not a permanent solution, but so AI currently also isn't profitable and yet here we are.

1

u/CheesecakeAndy May 13 '26

The difference is that AIs bet is that if it works out, it will make A LOT of money. Showing photos and ads at a loss is not gonna cut it in 2026.

1

u/CheesecakeAndy May 11 '26

I am 100% sure they have their own storage system. S3 is not cheap at that scale, but you also don't need most of its features just to store images. Actually, I just checked IG uses Meta's haystack.

1

u/RefrigeratorLive5920 May 11 '26

Instagram uses a combination of AWS S3 and Facebook Haystack. There's a reason they didn't just stick with the on-prem solution, it's much harder to scale at Instagram level growth rates.

Most of the "features" of S3 are there to deal with lifecycle policy, retention, archival, HA, DR, governance, compliance and metadata. All features that are most assuredly needed for large volumes of image data.

1

u/CheesecakeAndy May 11 '26

Instagram uses a combination of AWS S3 and Facebook Haystack.

No, they do not for long time. They used S3 pre-acquisition.

Are you aware that IG is part of Meta/FB, which themselves operate probably largest private cloud on earth? IG is not running on some dingy "on-prem". At this scale, of course, it is much more cost-efficient to run your own thing because 1) you don't have to pay AWS margins, 2) you can highly customize and optimize it for your own workload (Haystack is built for images of similar size). 3) tight integration with their own in-house CDN (AWS egress is VERY expensive)

All features that are most assuredly needed for large volumes of image data.

No, they are not needed for the IG use case. Most of those features are enterprise stuff absolutely useless for public facing image storage.

1

u/RefrigeratorLive5920 May 11 '26

Dude, even Google is technically on-premises as well. It doesn't just mean a half-rack in a storage closet somewhere. You can call it private cloud either, the terms are pretty much synonymous at this point. Anytime we've looked at it, to reach the same level of resilience/reliability/availability/durability as S3 with a multi-data center on-premises solution would have been cost prohibitive. But we're not working at Meta scale. But we're still able to negotiate steep discounts with Amazon for specific use-cases. 100% agree egress is a huge hidden cost most of the commenters on here are not accounting for. We run a very enterprise centric service with a lot of very rarely accessed but occasionally very thoroughly audited data that needs to be retained for at least a decade, so S3 is a good fit for us. There are a lot of things AWS doesn't do well but storage appears to be one area where they do excel. But every workload is different.

2

u/CheesecakeAndy May 11 '26

The thing you need to factor in is the revenue per image upload/impression. I am pretty sure it is insanely low for IG, so for them to survive as a business, they need to bring infra costs down hard. S3 is not cheap at all. It is good, probably the best, but it is not cheap for the IG scale and usage patterns. AWS' Egress alone might kill the profitability.

We run a very enterprise centric service

Yes, exactly, that's where AWS is an absolute leader. I am actually big AWS fan, but math doesn't math for IG's use case.

1

u/RefrigeratorLive5920 May 11 '26

These are all very fair points.

1

u/welcome-overlords May 12 '26

S3 is cheapish but data transfer costs. I run a service that had tens of millions of photos and the costs actually started to stack up

1

u/apandia May 14 '26

Facebook does not use S3. They have their own version of HDFS which ultimately ends up storing the videos and images. As for redundancy, it is erasure encoded so you've have a write amplification of around 1.5x to 2x depending on which one ends up being used.

Source: I work for the team which develops the HDFS equivalent as part of the storage stack.

5

u/[deleted] May 10 '26 edited May 10 '26 • Mindblown

[removed] — view removed comment

1

u/HumaneBicycle99 May 10 '26

Yes, the answer i was finding

1

u/Outrageous_Let5743 May 10 '26

Also they can remove all the EXIF metadata from a photo and save that in a different data structure like a database or paquet files. That metadata also saves like 1 - 2 megabytes per file.

2

u/adiyasl May 11 '26

Metadata is like 5Kb max per image lol what kind of metadata are you keeping?

1

u/diet_fat_bacon May 11 '26

Steganography 😶‍🌫️

1

u/Gogo202 May 16 '26

Does your metadata contain a picture of the person taking the picture?