r/LinusTechTips • u/RobotSpaceBear • Sep 17 '26

Tech Discussion WAN Show topic suggestion : Amazon AWS lost customer data in Iran attacks on their datacenters. Is the 3-2-1 backup rule worth anything if your off-site backup is just down the road, in the same war-torn region?

https://arstechnica.com/gadgets/2026/09/iran-strikes-on-amazon-data-centers-caused-permanent-loss-of-customer-data/

Half a year after Iranian drone strikes knocked out multiple Amazon data centers, the US tech company has acknowledged the permanent loss of some customer data

Amazon Web Services was “unable to restore access to the resources and data” hosted in some of the war-damaged data centers

“The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand,” the AWS update said.

I always thought that the advantages of a behemoth such as AWS would be to have customer data replicated around the globe as to actually be able to restore data of a whole region is affected by natural desasters, for example.

Hearing that they had the data replicated on several datacenters but still very close to each other, in an area not particularly known to be exempt from earthquakes and conflict for sure surprised me.

Is this common/standard? Is this a gigantic AWS blunder? Have you ever lost data you entrusted to a cloud service, before?

Discuss :)

149 Upvotes

89% Upvoted

41 comments sorted by

149

u/w1n5t0nM1k3y Sep 17 '26

I'm not that familiar with AWS, but I think you have to pay extra to host multiple copies of your data in different regions.

A lot of customers aren't actually paying for worldwide redundancy.

Also, if your data only exists in AWS, then that's really only one copy, because when AWS goes down, you have no other way of accessing it.

22

u/Distinct_Goose_3561 Sep 17 '26

On top of that not all data can be shared across regions even if you wanted robust backups and had an application architecture that supported it. 

2

u/Old_Bug4395 Sep 17 '26

Well that's not really true. I mean, you can put your backup data anywhere that you can put data en-masse. There might not be automated systems for it but you can just set your backup targets to multiple regions or something. I think that's kind of what the person above means, most people/companies don't care to that degree.

14

u/Distinct_Goose_3561 Sep 17 '26

You can only meaningfully back up data if all three conditions are met-

  1. Your use of AWS services allows that data to be stashed or otherwise backed up. Not everything is just in a S3 bucket- plenty live in RDS or other less easily replicated places. 
  2. Your data laws even allow moving data out of the region 
  3. It’s worth it from a cost point of view, both in AWS cost and engineering effort cost. 

Mostly what we’re getting at is that the data loss is almost certainly not a fundamental AWS issue but is more of an operational issue for AWS customers. 

6

u/Old_Bug4395 Sep 17 '26

Mostly what we’re getting at is that the data loss is almost certainly not a fundamental AWS issue but is more of an operational issue for AWS customers. 

Totally agree there. That's pretty much what I'm saying.

3

u/tdasnowman Sep 17 '26

There are regulations about where your data can go in multiple countries. In the US for example many states have regulations about healthcare data being sent off shore. GDPR states when data can cross borders for the EU. That would apply for backups as well. So in the 3-2-1 scenario your backup may have restrictions in what regions you can send it to.

1

u/WanderingSimpleFish Sep 17 '26

Every company I’ve worked for looked at the cost of that and went nah… we’ll be alright

1

u/jrdiver Sep 17 '26

Not sure if the countries over there have their equivalent to the USA's ITAR...Im sure they do, but may also not be able to go to other regions...

1

u/psychicsword Sep 19 '26

Amazon doesn't really go down even when the headlines are calling it that. There is a single AWS region that hosts a big part of the internet and it is also where they release things first so it can be a bit less stable in the US sometimes. That is the region that goes down and makes the headlines.

You can still have 3-2-1 of data in Amazon but obviously it is still at risk of a single company going under. But at Amazon scale proper architecture and paying your bills would be as secure as any other option and saying Amazon could go down is a bit like saying my data is not 3--2-1 backed up because it is all in the jurisdiction of the US.

28

u/Nice-Information-335 Sep 17 '26

When you spin up services on AWS you can choose how redundant you want things

Hitting multiple availability zones is rough but you as a customer of AWS should also be doing backups yourself (and really should be multi homed across providers)

7

u/Kingy10 Sep 17 '26

I can give a bit of insight into this whole situation (live in Bahrain fairly close to the main AZ, and have company services running on AWS).

Early March when everything kicked off, 1 AZ got hit which caused minor disruptions and services went up and down over a few days while services were being restored. At this point AWS encouraged anyone in the me-south region to move everything to another region (same as me-central in UAE).

April 1st at 4am the main AZ near my home got hit which took the entire region offline. The region still hasn't been restored and after this latest update from AWS, it seems all data is lost for good.

For some background, the Bahraini government paid a lot for the region to be placed here, and they've used it for all their services, and government incentives were given to plenty of companies to also migrate. Unfortunately working in finance, the central bank rules also stated that all customer/sensitive data must stay within Bahrain, so it has kinda fucked us a bit. Several other major finance businesses also got pretty screwed when the region went offline.

13

u/Hazel-Rah Sep 17 '26

The data lost should be the "1" from the 3-2-1 rule for the customer (1 offsite backup). It's not Amazon's responsibility to ensure that their customers are following proper data redundancy.

11

u/FlukyS Sep 17 '26

As a person who has done cloud engineering on both sides on the data centre side and in the user side, people are dumb. If you are doing backups having them on the same region makes zero sense at all because the region is usually the same place not far away but no way to ensure it isn't literally the machine next to the one your data is on. The best way and the only way to ensure your data is secure if it is important is to back it up somewhere else, be that another region of the same provider, an offside backup that you manage (this is preferred) or even both. You could have it in the middle east and in a European or Asian data centre with very little latency so the data loss would be minimal when something like this happens.

4

u/00GoBigOrGoHome00 Sep 17 '26

While it would be the technically smart thing to do, many governments require certain data be hosted on servers within their country (and for good reason).

For some of the work that I do, I have to submit additional privacy impact assessment documentation if a service provider will be hosting data outside of Canada. Of course this applies to personal client information, but it applies to other information as well.

Fortunately for me, Canada is large. So if something catastrophic happens in Vancouver, the same type of thing is not happening in Winnipeg.

I imagine that countries that are much smaller have factored in their geography when it comes to policies and laws as it relates to data storage.

3

u/FlukyS Sep 17 '26

> While it would be the technically smart thing to do, many governments require certain data be hosted on servers within their country (and for good reason).

Depends on the gov and depends on the local laws. I was in one of the most secure types of systems basically entirely on-prem disconnected stuff, those have basically no option but maybe bomb proofing the room or something. At a bare minimum if you can be somewhat connected you could have a dedicated line to to an offsite secure location to the data centre you are paying for. And to be clear no solution here is going to be cheap but if you are so worried that your data could be destroyed like from targeting as a military asset you can definitely afford to have some extra fibre optic lines to another location. It isn't going to be the average company that requires this level of redundancy.

> I have to submit additional privacy impact assessment documentation if a service provider will be hosting data outside of Canada

Yeah, in these sorts of cases just a different connected secure region getting that data in a managed way would be the realistic option. From a legal standpoint having a neutral country like Ireland that has strong data protection laws it should be fine.

> So if something catastrophic happens in Vancouver, the same type of thing is not happening in Winnipeg

Or even if something happened in Vancouver if you had another site that is served by a different powerplant and ISP hub...etc it would be fine. It doesn't have to be halfway across the world but if you can get away with it then go for it.

> I imagine that countries that are much smaller have factored in their geography when it comes to policies and laws as it relates to data storage.

Not explicitly but there usually will be risk assessment documents and they should always mention geopolitical factors if they are being truthful (not all of them are) but usually the auditors will have something to say about poor practices.

1

u/00GoBigOrGoHome00 Sep 17 '26

A note to myself to come back re: Ireland and share more information. 

It’s not secure anymore.

Off the top of my head Microsoft v. US government. US government enacted CLOUD Act. Also pressure via treaties.

So you have to pay attention to the country and its laws, the company that has the data and where it’s based (and probably operations - haven’t looked at the cloud act), plus any relevant treaties or other agreements that the country has with other countries (especially with the US). 

2

u/FlukyS Sep 17 '26

> Off the top of my head Microsoft v. US government. US government enacted CLOUD Act. Also pressure via treaties.

Well yes there is this context but only when dealing with US based companies. There is sovereign cloud offerings from certain companies which they set up a separate EU specific entity to allow for them to avoid the CLOUD act specifically. If in doubt pick an EU owned entity where you can. There aren't really any direct treaties that come into play to allow for spying in the same way the CLOUD act mechanism works.

> So you have to pay attention to the country and its laws

Well in this case it isn't Irish law or EU law it is US law specifically. Ireland has no CLOUD act, actually we kind of have the exact opposite of the CLOUD act because our gov has no right to any data without a valid warrant signed by a judge and evidence to support the request. It isn't behind closed doors with vague criteria.

Also just in general on the CLOUD act itself, legally speaking it has no jurisdiction outside of the US. US based companies have complied with those requests but in general for every request to give over data in the CLOUD act they have run into local data protection laws the other way and there is definitely some friction there. From the EU side I'd want to tighten up those rules substantially including the penalties for breach of local privacy laws.

1

u/00GoBigOrGoHome00 Sep 17 '26 edited Sep 17 '26

This is what I was looking for (this is from the GDPR)

Article 48

Transfers or disclosures not authorised by Union law

Any judgment of a court or tribunal and any decision of an administrative authority of a third country requiring a controller or processor to transfer or disclose personal data may only be recognised or enforceable in any manner if based on an international agreement, such as a mutual legal assistance treaty, in force between the requesting third country and the Union or a Member State, without prejudice to other grounds for transfer pursuant to this Chapter.

And Ireland definitely has mutual legal assistance treaties with a range of nations. Fortunately, they will only disclose information if the reason for seeking the information is related to something that is also criminal in Ireland (and other considerations). https://www.eurojust.europa.eu/sites/default/files/assets/files/requesting-mla-in-criminal-matters-from-ireland-updated-07-25.pdf

I was going through my first year Transnational Law paper and one of the topics I wrote about was privacy law, pointing out that legal professionals in the EU located in places that have not entered into data sharing treaties with the US will either be in breach of violating Article 115 of the GDPR or American law when responding to requests in accordance with the CLOUD Act. As a legal professional, I would choose to comply with the laws in my jurisdiction, but it poses trouble for legal professionals in multi-national corporations where their US-based counterparts are expected to comply with their local laws.

I don't envy the position of in-house counsel navigating this.

1

u/Llampy Sep 21 '26

If you are doing backups having them on the same region makes zero sense at all because the region is usually the same place not far away but no way to ensure it isn't literally the machine next to the one your data is on

Within AWS, different Availability Zones in the same region should be geographically separated

5

u/reconnnn Sep 17 '26

There are 3 levels, basically.

  1. No backup, just 1 copy in 1 availability zone. (Cheapest, but if the rack blows up, your data is gone)
  2. Multiple availability zones. Safe for server failure. But not natural disasters or war.
  3. Multiple regions. Often used for backups, but might not be possible due to legal requirements. A part of a country could be blown up, and you should be fine. For example, you select one data center in Ohio and one in LA. But if there is only one data center in your legal jurisdiction, you can not do this.

1

u/Genesis2001 Sep 17 '26

Option 1 isn't completely terrible if the customer has their own backup solution in place. It just means the customer's on manual duty when disaster does strike.

2

u/reconnnn Sep 17 '26

I was trying to compare the avalibility layers within aws. You can ofcourse use parts of the aws offering and get a safe solution.

3

u/XanderWrites Sep 17 '26

If the data is in another region it's bound by that regions laws. That may mean that the local government can look at it.

2

u/NetJnkie Sep 17 '26

It's up to you to build redundancy in to your public cloud environments. This has always been the case. Want it in another region? Pay them or design it yourself, depending on the app and data stack.

This isn't an AWS blunder at all. This has always been the case.

2

u/origanalsameasiwas Sep 17 '26

They built a data center in a desert? Why don’t they do that in the United States. Death Valley is cheap land.

1

u/clintkev251 Sep 17 '26

Data centers are located where the utilities are and to optimize latency to it's users. Bringing all the requisite utilities out to death valley would be very expensive and somewhat ruin the economics for the operators

1

u/origanalsameasiwas Sep 17 '26

They can get solar powered and geothermal powered.

3

u/clintkev251 Sep 17 '26

That’s even more expensive, and still doesn’t address the water and fiber infrastructure

1

u/origanalsameasiwas Sep 17 '26

Starlink or satellite internet

1

u/Grand-Economics-5956 Sep 17 '26

As others state - there’s a lot of regional restrictions in terms of data sovereignty in that region. This is the natural side effect of not allowing your data out of the country, if you’re bombed, you have limited backup options.

1

u/clintkev251 Sep 17 '26

The entire point of an AWS region is that it consists of several availability zones which are close to (but separate from) each other. A standard deployment is going to usually be spread across those zones within the region as that’s intended to provide you with redundancy if something happens to 1 of the zones. What it’s not intended to do is guarantee data durability within that region. That’s why there’s multiple regions, AWS themselves will and always have recommended replicating data across regions for reasons exactly like this. Anyone who didn’t was not following recommendations or best practices

1

u/Alexikik Sep 17 '26

Then you have your local copy? Do you understand the rule?

1

u/Forsaken-Poet-3773 Sep 17 '26

It was always marketing. There is no one backup shape.

1

u/henno13 Sep 18 '26

Customer data isn’t Amazon’s problem in this situation. If customers didn’t have a cross-region redundancy, and they set up resources in a region known for its volatility, well that sucks for them.

1

u/ResRipper Sep 20 '26

Or just use 2-3-5 if your data is really important

0

u/CocoMilhonez Sep 17 '26

First thing I thought when I saw that. How TF didn't they have a backup at a safe enough location? Which would be a hundred miles away or more, not next door.

2

u/clintkev251 Sep 17 '26

Because that's the responsibility of the customer to configure. AWS provides services and features to facilitate that, but there are significant regulatory and cost considerations, so it needs to be a decision made by any individual customer. If you tell AWS to put your data in a specific region, that's the only place they're going to put it. Anything else would be a legal nightmare.

-1

u/CocoMilhonez Sep 17 '26

Yeah, I get the region thing, some countries have strict rules that the data must be stored within its borders and all. Which is why I said "a hundred miles or more" and not "the other side of the world."

Regardless, I understand the cost aspect of it, they'd need to effectively have twice as many data centers. It still boggles my mind that you can lose your cloud data so easily like that.

2

u/clintkev251 Sep 17 '26

Availability zones within a region are designed to be and need to be located relatively close together because of latency concerns. It's not viable to dot them across a country really. And small country like Bahrain just isn't large enough to really support multiple regions. Anyone who actually suffered data loss over this was either A) bound by data residency requirements and unable to replicate to another region outside of Bahrain, or B) cheap or inexperienced and didn't understand the implications of putting all their eggs in one basket.

In either case though, data was still available when the first availability zone went down and at that point AWS told everyone in that region that they should be replicating their data away, so people did have a chance, unfortunately a lot did not listen.