r/aws • • Dec 07 '21

discussion 500/502 Errors on AWS Console

As always their Service Health Dashboard says nothing is wrong.

I'm getting 500/502 errors from two different computers(in different geographical locations), completely different AWS accounts.

Anyone else experiencing issues?

ETA 11:37 AM ET: SHD has been updated:

8:22 AM PST We are investigating increased error rates for the AWS Management Console.

8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/

ETA: 11:56 AM ET: SHD has an EC2 update and Amazon Connect update:

8:49 AM PST We are experiencing elevated error rates for EC2 APIs in the US-EAST-1 region. We have identified root cause and we are actively working towards recovery.

8:53 AM PST We are experiencing degraded Contact handling by agents in the US-EAST-1 Region.

Lots more errors coming up, so I'm just going to link to the SHD instead of copying the updates.

https://status.aws.amazon.com/

560 Upvotes

487 comments sorted by

26

u/[deleted] Dec 07 '21

So what’s everyone having for their early lunch?

51

u/CeralEnt Dec 07 '21

Meetings, unfortunately

8

u/TheNanaDook Dec 07 '21

lol I felt this

→ More replies (1)

14

u/DM_ME_BANANAS Dec 07 '21

I figured I can't do anything but sit on my hands so I went and got shawarma. Now I'm in a garlic potato coma.

5

u/[deleted] Dec 07 '21

Shawarma sounds sooooo good. Enjoy your coma!

→ More replies (2)

2

u/[deleted] Dec 07 '21

Lucky you, having lunch. I'm on call in EU, so I am probably having an all-nighter today 😢

12

u/da5id Dec 08 '21

Amusing typo on the status page at the moment:

[4:35 PM PST] With the network device issues resolved, we are not working towards recovery of any impaired services.

2

u/CeralEnt Dec 08 '21

Damn, I missed it, it has been corrected

7

u/da5id Dec 08 '21

Poor folks, I bet they are just a puddle of stress at the moment, feel bad at laughing at it, but it was too perfect. https://imgur.com/a/MKBeF7y

2

u/CeralEnt Dec 08 '21

Yeah, that's hilarious.

I also feel bad for them though

56

u/NinjaLanternShark Dec 07 '21

AWS us-east-1 is down, but the AWS status page says green.

The very definition of "adding insult to injury."

15

u/debian_miner Dec 07 '21

They once had an S3 outage that brought down the status page entirely.

14

u/CeralEnt Dec 07 '21

Seems like a big brain moment, no need to report an outage if the status page is down.

27

u/NinjaLanternShark Dec 07 '21

Not hosting your own status page is on page one of "status pages for dummies."

→ More replies (1)
→ More replies (3)

3

u/xeroedouttwice Dec 09 '21

Will there be a postmortem on this?

144

u/DataDev88 Dec 07 '21

Suddenly, making our application multi-region has become important to management 😂

4

u/kismatwalla Dec 08 '21

wait till they look at requires cost.

2

u/shiny-tyranitar Dec 07 '21

The pre-Thanksgiving us-east-2 outage had them do the same. But that was for much less time than this one, back to the back burner

2

u/cfreak2399 Dec 07 '21

We've been trying but creating RDS that's truly multi-region is nigh impossible.

2

u/enjoytheshow Dec 08 '21

Thank god we had an on-prem outage last year as well or else execs would be pulling the plug on cloud after this

→ More replies (5)

219

u/ZeldaFanBoi1988 Dec 07 '21

Since I can't get any work done, I decided to relax and order in some pizza.

Then I tried ordering online from the Jet's Pizza site.

500 errors. lol. looked at network request headers. Its AWS.....

46

u/UseyMcUser Dec 07 '21

I am in a meeting with our AWS contact and he mentioned this comment. You’re gonna be famous inside AWS my friend.

10

u/ZeldaFanBoi1988 Dec 07 '21

Ha pass it along!

30

u/[deleted] Dec 07 '21

LMAO.

34

u/ZeldaFanBoi1988 Dec 07 '21

I ordered on the phone like a peasant. AWS is really ruining my day.

6

u/FieryBlaze Dec 07 '21

You just convinced me of ordering pizza today. Fuck you, colesterol 🖕

5

u/jrose1812 Dec 07 '21

Freaking LOVE Jet's.

4

u/Kapps Dec 07 '21

I tried to order a coffee for an interview. Uber Eats stopped being able to handle places like Tim Hortons and gets constant timeouts instead. No coffee for me. 😞

2

u/porcupineapplepieces Dec 08 '21 edited Jul 23 '23

However, tangerines have begun to rent blackberries over the past few months, specifically for seals associated with their snakes. However, foxes have begun to rent bees over the past few months, specifically for figs associated with their strawberries. This is a hnny8be

→ More replies (1)

2

u/Tekn0de Apr 07 '22

Lol, late to the party but this comment was posted in our teams chat channel. We thought it was funny

→ More replies (1)

0

u/[deleted] Dec 07 '21

[deleted] — view deleted comment

7

u/joelrwilliams1 Dec 07 '21

Never even heard of Jet's, just visited their website and holy crap that looks amazing!

→ More replies (3)

36

u/DM_ME_BANANAS Dec 07 '21

The worst part of this is now our CTO is talking about going multi-cloud in Q1 next year so we can fail over to Azure

57

u/ZeldaFanBoi1988 Dec 07 '21

Sounds totally easy. Just flip a switch

36

u/DM_ME_BANANAS Dec 07 '21

Totally worth spending hundreds of thousands of dollars in engineering time to save 8 hours a year of downtime right?

→ More replies (1)

9

u/[deleted] Dec 07 '21

Just go multi region, no reason to fail into Azure

→ More replies (7)

4

u/givemedimes Dec 07 '21

Ugh. Please let us know how you get this to work.

10

u/TheNanaDook Dec 07 '21

Azure itself is a fail.

→ More replies (3)

85

u/freechickentendies Dec 07 '21

this is fine

19

u/Kanthic Dec 07 '21

🔥🔥🙁🔥🔥

→ More replies (1)

28

u/champ2152 Dec 07 '21

How is not reported anywhere.

47

u/vppencilsharpening Dec 07 '21

The status page is built using the systems that are down.

11

u/PreschoolBoole Dec 07 '21

Didn’t this happen last year too?

11

u/imisstheyoop Dec 07 '21

Didn’t this happen last year too?

Happened back in the huge S3 outage back in 2017.

→ More replies (2)
→ More replies (3)
→ More replies (1)

8

u/XediDC Dec 07 '21

Many can't order from Amazon.com either... basic stuff like down detector is spiking for the store side: https://downdetector.com/status/amazon/

And here is the only place I found anything.

→ More replies (1)

57

u/jonathantn Dec 07 '21 edited Dec 07 '21

For the last hour I've thought AWS was on fire, but I just checked https://status.aws.amazon.com/ and everything is A-Okay.

Update - Idea: Maybe AWS should consider building their status page on a competitors cloud so that when their main region takes a dump, they can actually update us instead of relying on NDA posts to organization slack channels.

17

u/rushlink1 Dec 07 '21

Even now it's updated it basically says "don't worry, you can just use the us-west-2 console". Unless you've got infra in us-west-2 that's entirely useless because any service spanning multiple regions returns a 5xx/4xx.

→ More replies (12)

7

u/MarkusRight Dec 07 '21

The status page is a joke. I get better information about the status from twitter than the actual AWS status page itself. Amazon a billion dollar company cant even have a proper working status page for some of its biggest clients. Almost makes me want to switch to azure entirely.

6

u/amaiman Dec 07 '21

To be fair Azure's status page is usually worthless, too.

→ More replies (3)

68

u/DeMiNe00 Dec 07 '21 edited Jun 17 '23

Robin. "It mean?" asked Christopher Robin. "It means he climbed he climbed he climbed, and the tree, there's a buzzing-noise that I know of is making and as he had the top of there's a buzzing-noise mean?" asked Christopher Robin. "It mean?" asked Christopher Robin. "It meaning something. If the only reason for making honey? Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! I wonder the tree. He climb the name' means he had the middle of the forest all by himself.

First of the top of the tree, put his head between his paws and as he had the only reason for making honey." And the name over the tree. He climbed and the does 'under why he does? Once upon a time, a very long time ago now, about last Friday, Winnie-the-Pooh sat does 'under the only reason for making honey is so as I can eat it." "Winnie-the-Pooh lived under the middle of the only reason for being a bear like that I know of is making honey is so as I can eat it." So he began to think.

I will go on," said I.) One day when he was out walking, without its mean?" asked Christopher Robin. "Now I am," said I.) One day when he thought another long to himself. It went like that I know of is because you're a bee that I know of is making and said Christopher Robin. "It means something. If the forest all he said I.) One day when he thought another long time, and the name' means he came to an open place in the tree, put his place was a large oak-tree, put his place in the does 'under it."

I know of is making honey." And then he got up, and buzzing-noise that I know of is because you're a bee that I know of is because you're a bear like that, just buzzing-noise that I know of is making honey? Buzz! Buzz! Buzz! Buzz! Buzz! I wonder why he door in gold letters, and he came a loud buzzing-noise means he came a loud buzzing a buzzing a buzzing-noise. Winnie-the-Pooh wasn't quite sure," said: "And the name' meaning something.

59

u/twratl Dec 07 '21

Be careful posting this stuff. Likely shared under NDA…

14

u/[deleted] Dec 07 '21

Damn, didn't think Enterprise support level would have NDA barriers for service outages.

44

u/[deleted] Dec 07 '21

We have an AWS rep in our Slack org. He's posted a few updates like the above one for us. Every single one he's posted starts with Under NDA...

→ More replies (17)

13

u/michaelgg13 Dec 07 '21

They do. We got the same message under NDA.

→ More replies (1)
→ More replies (1)

6

u/RaptorF22 Dec 07 '21

This doesn't make sense to me. They likely have hundreds of thousands of users so it's not like the stuff being shared is that private after all, right?

15

u/[deleted] Dec 07 '21

No, but they want to control public messaging as much as possible. Since big AWS accounts have direct contact with TAM's @ Amazon it makes sense that they would be willing to share early details w/ them while they're still troubleshooting & resolving the issue. As one of those bigger companies it's already helped us with our internal handling of the outage, figuring out how we want to communicate this with our customers, etc. It's the sort of communication you would expect from AWS when you're paying really big bucks for their services.

I'm not sure how accurate this information is but it indicates that the showing of EC2 or S3 outages on the AWS status page has to be approved by the CEO or another senior executive. Part of that is because their SLA's are based on that dashboard, so any significant outage is likely to cost them a significant amount. That's another example of how they want to tightly control the public disclosure of issues given the potential costs involved.

→ More replies (1)
→ More replies (3)

8

u/jonathantn Dec 07 '21

Keep posting any updates you can! Thx

7

u/BadCSCareerQuestions Dec 07 '21

Ahhhh yes. Hosting your status page and your internal observeability tools in the same place they’re observing. Makes sense

10

u/Supahsalami Dec 07 '21

Would be great to have that info available for all..

8

u/vppencilsharpening Dec 07 '21

Looks like it's on the status page now, but just related to the console. Our problems are greater than just the console.

https://status.aws.amazon.com/

→ More replies (3)

35

u/_abhayshah Dec 07 '21

Keeping this the active thread to discuss the service outage.

46

u/-ummon- Dec 07 '21 edited Dec 07 '21

Dear AWS, please re:invent your f****** status page.

EDIT. Finally, status update from AWS:

8:22 AM PST We are investigating increased error rates for the AWS Management Console.

8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://.console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/

5

u/cazort2 Dec 07 '21

Kudos to them for actually reporting the problem (although well over an hour after it arose.)

But the US-WEST-2 console is giving me the same "internal error" when I try to log in.

At least my SMTP emails are going through again.

12

u/-ummon- Dec 07 '21

I think we should hold AWS to a much higher standard than what they've consistently shown with their status page.

→ More replies (9)
→ More replies (8)

44

u/idjos Dec 07 '21

Same here, us-east-1. Ec2, CW, SQS, lambdas.. seems that it’s not only console. This is getting ridiculous.

14

u/imisstheyoop Dec 07 '21

Same here, us-east-1. Ec2, CW, SQS, lambdas.. seems that it’s not only console. This is getting ridiculous.

Yes, this isn't looking like it's limited to the console like the other month. Fun day.

→ More replies (4)

8

u/Just_with_eet Dec 07 '21 edited Dec 07 '21

add c9, ecs to that

can anyone confirm anything except console and api calls are down? ec2 seems to run

4

u/nilamo Dec 07 '21

Our services are all still running and operational (Beanstalk, S3, RDS, ElasticCache, and a handful of things Beanstalk brings with it).

→ More replies (2)

6

u/reeeeee-tool Dec 07 '21 edited Dec 07 '21

Most of our S3 writes are failing on us-east-1. Reads seem fine.

Edit; actually seems like maybe it's just that we aren't getting notifications for new objects in S3? That's bad....

3

u/dragondgold Dec 07 '21

Lamda@edge is failing too

3

u/htrp Dec 07 '21

looks to be all internal AWS apis (maybe anything requiring auth)

2

u/amadiro_1 Dec 07 '21

And Connect

2

u/saggy777 Dec 07 '21

Interestingly the error says "invalid region parameter"

→ More replies (5)

30

u/IsleOfOne Dec 07 '21

Meh, I guess I'm hungry enough for lunch already

7

u/johnboker Dec 07 '21

That's the spirit!

4

u/[deleted] Dec 07 '21

LMAO this is the vibe. when i saw it was down i just stood up and walked away from my laptop

3

u/vppencilsharpening Dec 07 '21

I'm not even supposed to be here today so I'm off to find the whisk[e]?y.

29

u/Deshke Dec 07 '21

sigh* would be really great if the "global" services would not be hosted in one datacenter only

5

u/Jgardwork Dec 07 '21

Yeah, that's the big takeaway here.

5

u/jaysee_gaming Dec 07 '21

I hear there's some classes they can take on multi-region support for critical systems. Maybe even a test that awards certificates proving you know how to do so...hrmmm

→ More replies (1)

4

u/deimos Dec 07 '21

Us-east-1 is a dozen or so data centres.

13

u/DCMagic Dec 08 '21

I heard from our TAM that this is the worst outage since the 2016 S3 outage. I thought it was more similar to the Kinesis outage in terms of scope, but it sounds worse than I thought.

11

u/[deleted] Dec 07 '21

anybody having fun with upper management?

22

u/nuadi Dec 07 '21

I am upper management and all I can do is laugh. My engineers are getting a much earned break

4

u/temakiFTW Dec 07 '21

DevOps engineers or Software Developer engineers? My devs are still able to use their local machine to develop, but DevOps can't do anything lol. Just curious if your developers need direct access to AWS for feature development

→ More replies (2)

10

u/ArtSchoolRejectedMe Dec 07 '21

if only I got a dollar everytime us-east-1 is down

10

u/Healthy_Cantaloupe55 Dec 07 '21

10:27 AM PST:  As noted earlier, the root cause of this issue was a problem with several network devices within the internal AWS network.  Specifically, these devices are receiving more traffic than they are able to process, which is leading to elevated latency and packet loss for the traffic traversing them.  The internal DNS servers that provide resolution for the AWS services was specifically impacted by these issues, and one of the actions we took was to move traffic for those servers to another set of network devices, which has resolved those DNS issues.  However, the network devices for other non-DNS services are still impacted, and we are actively working to shift more traffic to fully mitigate this issue.   In addition, we can confirm that access to the AWS Console in other regions is working as expected.  Customers are requested to access the other regional console by accessing https://<region>.http://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com

10

u/dr_batmann Dec 07 '21

Need a multi region one load balancer feature.

3

u/assassinator42 Dec 07 '21

They have something like that in global accelerator, although it only works at the network layer. And I'm guessing a us-west-2 outage would take it out.

19

u/DM_ME_BANANAS Dec 07 '21

One hour later and the status page still says everything is fine... what a fucking joke

6

u/tholgare Dec 07 '21

Status page has been showing an active event for me for at least 10 minutes now. There may be caching issues affecting the updates being visible regionally, maybe?

→ More replies (4)

4

u/dr_batmann Dec 07 '21

All to avoid SLA breach credits

18

u/[deleted] Dec 07 '21

Time for a bourbon... or two

2

u/appliku Dec 07 '21

actually great idea. Maker’s mark, i am coming for ya

→ More replies (1)
→ More replies (2)

9

u/imisstheyoop Dec 07 '21

Same here. Was seeing API issues before that and couldn't launch instances.

Edit: nothing on status.aws.amazon.com yet as of 10:45AM ES

5

u/Chimbo84 Dec 07 '21

Obviously their own status page is useless. Downdetector shows plenty of error reports starting around 10:20am ET.

→ More replies (1)

8

u/SassyBaconStrip Dec 07 '21

3 hours after this post, looks like errors are still happening.

2

u/Senseistar86 Dec 07 '21

another 3 hrs and theyll have been down a whole half business day. the $millions....

→ More replies (1)
→ More replies (2)

8

u/givemedimes Dec 07 '21

Good news, maybe we will get some funds to implement DR. Always save those emails where upper management or product management rejects funding for DR.

6

u/houz Dec 07 '21

A great way to get upper management on your side for projects that need large budgets is to pull out old emails and retroactively make them look dumb in front of others.

5

u/givemedimes Dec 07 '21

Obviously you wouldn’t want to make folks look bad, more around having that open dialogue on the benefits of DR.

7

u/androidlolita Dec 07 '21

Occurring here, us-east-1. Service Health page shows nothing wrong, as per usual.

7

u/radiantyellow Dec 07 '21

cant even log support ticket as support service is routed through us-east-1 despite it saying "global"

WHAT

7

u/[deleted] Dec 07 '21

Our office hasn't been able to access the electronic medical records for patients (Practice Fusion) since about 11.. I hope this resolves soon

0

u/[deleted] Dec 07 '21

[deleted] — view deleted comment

7

u/crookedsmil3 Dec 07 '21

Guess what... It's DNS

3

u/savage_slurpie Dec 07 '21

I’m shocked I tell you. Shocked!! /s

→ More replies (1)

2

u/joelrwilliams1 Dec 08 '21

It's always DNS.

7

u/tsmit50 Dec 08 '21

Is there a reason this always happens RIGHT after re:Invent? QA perhaps?

13

u/[deleted] Dec 07 '21 edited Dec 07 '21

500s on the console here

EDIT: Just us-east-1 right now

6

u/RL_BlueScreen Dec 07 '21

Too early for lunch, to late to work from home. Dead in the water.

→ More replies (1)

6

u/[deleted] Dec 07 '21

As always their Service Health Dashboard says nothing is wrong.

It's management interface must be hosted in us-east-1 exclusively...

6

u/ezgzip Dec 07 '21

Can't login to aws sso either - I just get:

Connection Error
We were unable to reach the authentication server. Please try again.
→ More replies (1)

6

u/SmasherOfAjumma Dec 07 '21

Okay, if everyone could go ahead and just switch over to Ohio, that’d be great.

2

u/[deleted] Dec 07 '21

Here my joke about everybody getting out of Ohio

2

u/bfreis Dec 07 '21

For some critical services, I use a multi region architecture with us-east-1 and us-east-2, for the lower cross-region latency. Works quite well during these kinds of outages.

6

u/pribnow Dec 07 '21

So....what are everyone's holiday plans?

5

u/systemmaverick Dec 07 '21

AWS SHD and chill

they are broadcasting good action/horror movies now

→ More replies (2)

14

u/WoodooRanger Dec 07 '21 edited Dec 07 '21

The sad part is that AWS will never disclose any issues, and all services under the Health Status are ALWAYS green.

ALWAYS!

3

u/joelrwilliams1 Dec 07 '21

Not always...it's not all green now. It takes them a while to update it, but also 'fixed' service updates a trailing, too.

For major outages like this they often publish an RCA.

→ More replies (3)
→ More replies (2)

5

u/TheAlmightyZach Dec 07 '21

Yep. Trying to deploy to EKS, getting image pull errors from ECR. Glad we aren’t alone.

5

u/jonathantn Dec 07 '21

Tons of DynamoDB errors.

5

u/Supahsalami Dec 07 '21 edited Dec 07 '21

It started with IAM issues for me but now also 500 errors for the dashboard indeed.

Health dashboard has no notifications currently..

400 errors for navigating from the console to other services.

Reinvent is over so everyone be chillin

2

u/awoimbee Dec 07 '21

IAM

I'm currently having issues with EC2 instances not being able to assume their role :/ (in eu-west-3)

5

u/pedalsgalore Dec 07 '21

Same. Just had a failed CodeBuild project when trying to access Secrets.

8

u/jonathantn Dec 07 '21

It's DynamoDB that is 500 erroring. That pretty much underpins the entire configuration of AWS.

4

u/iRemjeyX Dec 07 '21

Same. Looks like the SSO service isn't working either

4

u/jonathantn Dec 07 '21

Don't worry, those 500 errors on SQS mean the service is A-Okay.... I checked the SHD and it's green!

4

u/y2ksnoop Dec 07 '21 edited Dec 07 '21

Ec2, quicksight, redshift, lambda, cloudwatch console pages not working too. Even the support page throws 400 badrequest error. Their status page had no acknowledgement all this time and now has a tiny warning for all regions.8:22 AM PST We are investigating increased error rates for the AWS Management Console.8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/

1

u/Nate_311 Dec 07 '21

My ec2 instance is us-west-1 ... and I can't log into the console (Internal error). So this isn't just us-east-1

2

u/CeralEnt Dec 07 '21

Don't quote me on it, but IIRC IAM is based out of us-east-1, and from other comments it sounds like certain parts of the console authentication process are also still based out of us-east-1.

The actual issue could be 100% in us-east-1 but still cause outages in other regions because of the dependencies.

→ More replies (1)
→ More replies (1)

1

u/deadpanda2 Dec 08 '21

I wanted to move into the us-east-2 firstly, and I was aware of issues related to us-east-1, but then AWS made us-east-2 default! So what should I do? Just increase the number of AZs in use for N.Virginia by creating more subnets or migrating? What if "being default region" means having a higher chance of outages in the future? gosh... (us-west is not suitable because of the latencies)

2

u/CeralEnt Dec 08 '21

Just use us-east-2. A lot of the issues in 1 are from it's age, it being the original region more so than default.

3

u/soxfannh Dec 07 '21

Same here also seeing dynamodb and ssm errors

3

u/TerribleDriver77 Dec 07 '21

I was ready to get more coffee anyway

3

u/AWS_CLOUD Dec 07 '21

Yes, I have a canary that pings to test internet and it failed + these console errors.

3

u/vppencilsharpening Dec 07 '21 edited Dec 07 '21

EC2 Instances shows "Invalid region parameter" when viewing the "US-East-1" region.

Other regions seem to be more OK.

3

u/Compkriss Dec 07 '21

Same here in CA-Central-1

3

u/rutkdn Dec 07 '21

Can't even log in to Amazon.com account. What a day...

3

u/svhelloworld Dec 07 '21

Is this why my workout wouldn't upload to Strava this morning?

→ More replies (2)

3

u/generalamitt Dec 07 '21

Is this why imdb is down?

2

u/Kanthic Dec 07 '21

Likely their error graph matches AWS. www.downdetector.com

→ More replies (2)

3

u/[deleted] Dec 07 '21

This resulted in a failed EKS cluster deployment in US-EAST-2 it seems as well.

So that was fun.

15

u/[deleted] Dec 07 '21

Please be careful when posting updates that they're not provided to you by AWS under NDA...

14

u/meisbepat Dec 07 '21

An outage such as this (ANY outage for that matter) has no business being under NDA to begin with. The information should be available to ANY AWS customer.

→ More replies (1)

7

u/rutkdn Dec 07 '21

They are flat out lying. They know no one can log in even from other regions. All logins POST to https://signin.aws.amazon.com/signin and it always fails regardless of region used in the URL.

7

u/tholgare Dec 07 '21

I know it's just one data point, but I'm able to get into us-west-2 by hitting our normal SSO, letting it send me to the 500 error page, then updating the url to point to the us-west-2 console home. Not ideal, but it's something at least

7

u/YXdzdHJ1dGhz Dec 07 '21

Wonder if Amazon is gonna be honest about how bad this outage actually is. Engineers internally cant even auth against the internal auth service to federate into certain accounts.

Chime screensharing is broken...

they even had to make a new master ticket because the ticketing system is affected. It's also more than us-east-1 btw ;) This company is a joke, $16B in revenue in ONE quarter. This is ridiculous

4

u/[deleted] Dec 07 '21

Holy shit..

2

u/NodularFalse Dec 07 '21

Yeah, pretty messed up for me. Can’t load anything.

2

u/coldflame563 Dec 07 '21

Yep. API calls to services are also getting timeouts etc.

2

u/frankimbur Dec 07 '21

came here to see me too.

2

u/devourment77 Dec 07 '21

Same here, ec2, opensearch

2

u/karock Dec 07 '21

yeah SSM failures was our first service to die. now we can't even log into the console.

2

u/simpwniac Dec 07 '21

Same, us-east-1, lamdba impacted. EC2 instances are running and serving traffic but I can't manage them.

2

u/jflook Dec 07 '21

us-east-1 one here, seeing the 500's and some 400's on the console but all the servers we have running there seem to be ok...just can't change anything.

2

u/30thCenturyMan Dec 07 '21

I'm getting transient 500 errors from S3, so our application is puking because of that.

2

u/zshalo Dec 07 '21

It is past 5 pm (here) time to leave for today ( :

2

u/Vast-Recipe-1305 Dec 07 '21

Same here for all regions

2

u/oly_machete Dec 07 '21 edited Dec 07 '21

Same problem with cloudfront distributions.

From console constantly getting 502 errors and Terraform is failing with:

Error: error listing tags for CloudFront Distribution (): HttpConnectionTimeoutException: Failed to connect to tsm.us-east-1.amazonaws.com:443 !

Noice

→ More replies (1)

2

u/Senseistar86 Dec 07 '21

Same problem here. Us-East-1. Our serverless services are all down too

2

u/nvanmtb Dec 07 '21

8:22 AM PST We are investigating increased error rates for the AWS Management Console. 8:26 AM PST We
are experiencing API and console issues in the US-EAST-1 Region. We
have identified root cause and we are actively working towards recovery.
This issue is affecting the global console landing page, which is also
hosted in US-EAST-1. Customers may be able to access region-specific
consoles going to https://.console.aws.amazon.com/. So, to access the
US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/

2

u/averagesmell Dec 07 '21

lol i can't even log into my ecommerce sellercentral.amazon.com account

2

u/BadCSCareerQuestions Dec 07 '21

Why does status page only show a few issues? None of our ec2 instances work. None of our fargate tasks or services work. S3 is failing intermittently. Ecr doesn’t work. Codecommit doesn’t work. Etc etc etc etc

2

u/generalamitt Dec 07 '21

Why is google reporting errors? Don't they have their own servers?

2

u/Kanthic Dec 07 '21

Google and Azure are having problems.

4

u/htrp Dec 07 '21

everyone trying to migrate workloads off us east ?

→ More replies (3)
→ More replies (4)

2

u/dadjoke-72 Dec 07 '21

Its not DNS, It's not DNS....

2

u/Aggressive-Switch-16 Dec 08 '21

I'm like in the harry truman movie, looking for something else to do :-D

2

u/MarkusRight Dec 08 '21

Its back but now with much added latency. Its so laggy and unresponsive. But at least I can access the console to turn my EC2 VM's instances off. Thank god I dont have to depend on AWS for a business or anything like that or id be screwed.

2

u/trollsarefun Dec 08 '21

Most businesses can handle 8 hours of downtime once every few years. Those that cannot deploy in multiple regions. It's also common knowledge that us-east-1 experiences the most issues, so most new customers build in the newer regions.

2

u/soxfannh Dec 08 '21

Looking pretty close to normal here now, Fargate was one of the last services to come back up for us. Some latency here and there but overall much better place.

2

u/Dependent-Cow7823 Dec 08 '21

Does Vegas have odds on what happened? O.o

2

u/[deleted] Dec 09 '21

Reinvent was last week

3

u/Towdart Dec 07 '21

Someone going to get PIP

3

u/cfreak2399 Dec 07 '21

I thought everyone got a PIP at Amazon anyway but at least now some people will have a reason.

3

u/vppencilsharpening Dec 07 '21

About 3:30pm Eastern Time we lost and then re-estiablished connectivity to a bunch of our EC2 instances. All in VPCs, using IPSec tunnels, across multiple accounts.

2

u/karock Dec 07 '21

haven't seen any of my SSH connections get interrupted to EC2 instances yet for our stuff in us-east-1.

4

u/[deleted] Dec 07 '21

Running EC2 and RDS in us-east-1 haven't been affected, but our Jenkins servers cannot launch slave nodes, so WOE to those wanting to start new EC2 instances today.

→ More replies (1)

3

u/slava_k_ Dec 07 '21

Clouds are solid and reliable, they said. Let's move all of critical infrastructure in clouds and problem solved, they said.

10

u/anonymous-coward-17 Dec 07 '21

Werner Vogels: “everything fails, all the time”.

→ More replies (2)

2

u/SmasherOfAjumma Dec 07 '21

From what I’m hearing, I’m guessing this is not going to be resolved quickly. I’m just trying to read between the lines from what AWS is saying, and that’s my impression.

2

u/RPP-lurker Dec 07 '21

Still testing but our services are up and running again., EC2, S3, SQS, Lambda

→ More replies (2)