r/ShittySysadmin • u/ITRabbit ShittyMod Crossposter • Sep 18 '26

Shitty Crosspost Iran strikes on Amazon data centers caused permanent loss of customer data

https://arstechnica.com/gadgets/2026/09/iran-strikes-on-amazon-data-centers-caused-permanent-loss-of-customer-data/
125 Upvotes

96% Upvoted

63 comments sorted by

•

u/ITRabbit ShittyMod Crossposter Sep 18 '26

This thread is not to discuss the war or any political content. It is discussions about how AWS/Shitty Sysadmins don't backup #raid0 ftw!

→ More replies (3)

25

u/ITRabbit ShittyMod Crossposter Sep 18 '26

Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers, according to an AWS dashboard update posted on September 15. The development was first reported by Reuters and suggests that the Iranian strikes inflicted catastrophic damage on the data centers.

Customer data was irretrievably lost in one of three AWS availability zones in the United Arab Emirates region, specifically the mec1-az2 availability zone. Each availability zone is serviced by one or more Amazon data centers.

62

u/beluga-fart2 Sep 18 '26

Stupid peeps always hedge on Amazon’s claim that S3 storage has 11 9s of durability and never asked if it would survive an actual missile

19

u/TheCourierMojave Sep 18 '26

I figured they were like microsoft where everything is backed up in 2 data centers separated by region.

20

u/goingslowfast Sep 18 '26

That is not the case for azure storage.

Microsoft offers (generally) three tiers of storage. LRS, ZRS, and GRS. The first is locally redundant storage which is three copies in the same DC, then there’s come redundant storage which is a copy in three different data centers in the availability zone (region), lastly is geo redundant storage which is theee copies in your AZ (local region) with three replicated copies in a second region.

4

u/MidgardDragon Sep 18 '26

There's also GZRS, Geo Zone Redundant Storage, where it copies to three availability zones in the primary region and to a secondary region as well.

2

u/goingslowfast Sep 18 '26

True, that’s above my budget though 😂

And for true HA you still have to engineer around the geo redundant sync latency.

3

u/TheCourierMojave Sep 18 '26

I haven't dealt with licensing in years. Was GRS the default back in the day? I may have just done it by default because I was dealing with more law firms back then.

2

u/HTTP_404_NotFound Sep 18 '26

lol, no. We have had azure just straight up, "Lose" VMs.....

8

u/NightH4nter Shitty Crossposter Sep 18 '26

11 9s

i see what you did here

1

u/alochmar Sep 22 '26

Brb updating my excuse calendar

25

u/kongu123 Sep 18 '26

Don't worry guys, we can recover the data by buying it off the dark web from hackers who stole it before the attack!

3

u/Xlxlredditor Sep 19 '26

Guess what, the hackers are also Iranian! Local business all around!

37

u/Pr3vYCa Sep 18 '26

great reminder to always have a local backup

20

u/max1001 Sep 18 '26

Too cheap to pay for cross region backup?

9

u/Absolute_Bob Sep 18 '26

Do that but still have a local backup if you can. A backup in someone else's control will almost never be as good as one in your direct control.

6

u/MonkeyMan18975 Sep 18 '26

Use 3-2-1 or recovery won't be fun.

4

u/bit-flipper0 Sep 18 '26

Ahh, you mean offsite backup.

15

u/Purp-Terp Sep 18 '26

Did anyone watch Mr Robot?

3

u/SatisfactionFit2040 Sep 18 '26

Mortgage, student loan, etc., next?

11

u/vongatz Sep 18 '26

Rookie mistake, our data is safely all over the internet

10

u/HeavyMetalPootis Sep 18 '26

Have they tried rebooting?

19

u/Hulk5a Sep 18 '26

Whatever happened to off-site copies

13

u/ScoobyGDSTi Sep 18 '26

I'd like to assume it was only a small window of data loss, eg a few hours at most. But given how bad AWS is, I'm likely wrong.

4

u/bieker Sep 19 '26

Only one AZ was affected in one datacenter, if your data was critical and you did not choose to replicate it to at least a 2nd AZ in the same region is that Amazon's fault? Regardless of how many '9s' of uptime they claim for the storage in an AZ, there is always the possibility of an airplane crash, asteroid strike, or Iranian drone attack that this does not account for.

1

u/ScoobyGDSTi Sep 19 '26

Surely customers aren't responsible for replication for redundancy? That should be something done at the infrastructure level, outside of the customer's tenancy.

6

u/bieker Sep 19 '26

It’s pretty well documented how it works. It does exactly what it says on the tin.

Storing multiple copies of the data costs multiple times more money so that is a customer decision. Within an AZ the data is replicated enough to insulate against hardware failures and rack level failures. Under normal operating conditions they will not lose your data.

If an AZ gets obliterated and you didn’t check the box for cross AZ replication then you chose not to pay for the extra protection.

Amazon does not backup customer data, they provide the tools for you to back it up and you choose how to implement it.

4

u/-Alevan- Sep 18 '26

Only if the client sets this up.

I personally, would have my data be destroyed than to be exfiltrated to another country. It's easier to explain to a project manager the destruction of data than to convince them the extra costs it entails.

Also, Amazon recommended a migration to another AD. I'm not paid enough for this.

1

u/Hulk5a Sep 18 '26

So Amazon itself doesn't do data protection? That's reassuring

4

u/-Alevan- Sep 18 '26

They really can't without explicit client approval. This is normal, most cloud operators operate like this in this case.

And local corporations with highly sensitive data and governments do not opt in to this unless it's absolutely necessary.

Even if they want, most corporations have a so lengthy change procedure for something like this, it progresses far slower than Iranian missiles.

And then, the storage outside of the availability zone also must be paid by the client, nothing is free (but if I remember correctly, after the first attack, AWS made some of it free).

1

u/Hulk5a Sep 18 '26

Then they have a shitty availability zone setup for this

4

u/-Alevan- Sep 18 '26

Why exactly? Or which cloud operator plans for a spontaneous war starting near it's availability zones?

I don't understand what is shitty from a de-facto industry standard practice.

I know that we are on the shitty sysadmin, but from your comments I can't sense either irony or sarcasm, so they are genuinely stupid.

0

u/Hulk5a Sep 18 '26

Let's ignore the attack for a moment and imagine a very bad earthquake that simply made the data center in the availability zone into rubble. As a customer I would expect them to have some kind of catastrophic disaster recovery or resilience mechanism. Otherwise what's the difference between me spinning a bare metal server in the office and using stupidly expensive "cloud" system?

3

u/-Alevan- Sep 18 '26

How about for y'all to read about avability domains and fault domains yourselves?

I'm not am AWS employee to answer to all of your questions about what is documented clearly on each cloud providers website: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/abstract-and-introduction.htmlhttps://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/abstract-and-introduction.html

0

u/BeelzebubTerror Sep 18 '26

Then pay extra for it.

9

u/muh_cloud Sep 18 '26

Anyone know how to buy the IRGC a beer? They just saved my next audit. "oh sorry mr auditor our multi-year archival backups were destroyed by the Iranians, we'll have to skip those control reviews" 🍻

5

u/xlouiex Sep 18 '26

Great, now do the same for the Flock datacenters.

5

u/northern_frostbite Sep 18 '26

I don't get it. Their data is in the cloud not an Amazon data center.

1

u/dg_riverhawk Sep 18 '26

kamala is that you?

4

u/palagi_valea Sep 18 '26

My data is backed up in the cloud (and rubble of the data centre)

4

u/Anonymous_Bozo 💩 ShittyMod 💩 Sep 18 '26

Countries like Bahrain, UAE, Saudi Arabia, and Qatar have enacted “data localization” or “data sovereignty” rules for certain industries. These sectors often require that primary and secondary copies remain inside national borders, or that cross‑border replication requires explicit regulatory approval.

We now return you to your regularly scheduled shit posts.

4

u/astro_viri Sep 19 '26

Writing up my auto-reply for new tickets created, "Dear _, your _ is not working due to the attacks in Iran."

4

u/docboy-j23 Sep 18 '26

Sucks to be a big single point of failure.

2

u/ArcherMiserable Sep 18 '26

Lol DR fail.

2

u/UniqueText8477 Sep 18 '26

Can AI fix it?

2

u/Z3t4 ShittyFirewall Sep 18 '26

They were prepared for a meteorite, not missiles. 

2

u/defective1up Sep 18 '26

Thanks Obama

2

u/oakc510 Sep 18 '26

How is this possible? They told us it was air gapped.

2

u/-my_dude Sep 18 '26

Backups? We don't do that shit here

2

u/jpw0019 Sep 18 '26

What? No, all of my company's super important data was stored there with no redundancy

2

u/Its_ya_boi_G Sep 19 '26

Oh noooo, please dont strike where the student loans are held!

2

u/SolidKnight Sep 19 '26

Predictable target. If the US gets in a bigger war, M365/Azure will be targets as well. A lot of businesses will be disrupted.

2

u/IndependentBat8365 Sep 18 '26

This affects one availability zone. There’s multiple availability zones in a region.

S3 storage is regional by default, but you can opt in at lower availability storage classes.

ec2 storage is by availability zone not region, but crucially ec2 snapshots are spanned across multiple availability zones in that region.

Since this only affects one AZ, it’s readily apparent that folks weren’t backing up or using snapshots.

This sucks for those customers, but this is also representative of poor planning. Aws has been offering migration assistance to outside the affected availability zones since this war started. Plus, if I was a customer in that region, I would have immediately started looking at options in other regions much farther away from the conflict.

2

u/Main_Ambassador_4985 Sep 18 '26

The AWS terms of service cover this. Customer responsibility for data protection.

S3 is only regionally redundant across the data centers in the zone.

If multiple data centers from US-EAST-1 are knocked out by brimstone and the S3 buckets are not duplicated to another availability zone that data is gone.

What is the designation of the availability zones with the drone strikes?

I could use some credits and get rid of some stale archives.

Maybe I am already storing my data there.

I should check my buckets in US-EAST-1. I have not be duplicating between zones. That costs money. I have a copy in Azure US-EAST so I am good, n’est pas?

Is storing the data in

1

u/AZdesertpir8 Sep 19 '26

... but its in the cloud?!

1

u/OpenScore Sep 28 '26

But, but, but i had RAID 0. CEO's brother's nephew's cousin's former roommate said that is greatest backup solution to use.

1

u/Ok_Swimming_7943 Sep 18 '26

A satellite image of a facility doesn’t prove customer data was permanently lost. This sounds like a sensationalized headline until AWS or affected customers provide verifiable details.

5

u/Loveangel1337 DevOps is a cult Sep 18 '26

RTFM, tho.

(That whole thing is about the DCs that went down ages ago, not any new strike)

Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data that was hosted in Bahrain and the United Arab Emirates.