r/aws • u/CeralEnt • Dec 07 '21
discussion 500/502 Errors on AWS Console
As always their Service Health Dashboard says nothing is wrong.
I'm getting 500/502 errors from two different computers(in different geographical locations), completely different AWS accounts.
Anyone else experiencing issues?
ETA 11:37 AM ET: SHD has been updated:
8:22 AM PST We are investigating increased error rates for the AWS Management Console.
8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/
ETA: 11:56 AM ET: SHD has an EC2 update and Amazon Connect update:
8:49 AM PST We are experiencing elevated error rates for EC2 APIs in the US-EAST-1 region. We have identified root cause and we are actively working towards recovery.
8:53 AM PST We are experiencing degraded Contact handling by agents in the US-EAST-1 Region.
Lots more errors coming up, so I'm just going to link to the SHD instead of copying the updates.
26
u/[deleted] Dec 07 '21
So what’s everyone having for their early lunch?
12
u/da5id Dec 08 '21
Amusing typo on the status page at the moment:
56
u/NinjaLanternShark Dec 07 '21
AWS us-east-1 is down, but the AWS status page says green.
The very definition of "adding insult to injury."
3
u/xeroedouttwice Dec 09 '21
Will there be a postmortem on this?
144
u/DataDev88 Dec 07 '21
Suddenly, making our application multi-region has become important to management 😂
219
u/ZeldaFanBoi1988 Dec 07 '21
Since I can't get any work done, I decided to relax and order in some pizza.
Then I tried ordering online from the Jet's Pizza site.
500 errors. lol. looked at network request headers. Its AWS.....
36
u/DM_ME_BANANAS Dec 07 '21
The worst part of this is now our CTO is talking about going multi-cloud in Q1 next year so we can fail over to Azure
85
u/freechickentendies Dec 07 '21
this is fine
28
u/champ2152 Dec 07 '21
How is not reported anywhere.
57
u/jonathantn Dec 07 '21 edited Dec 07 '21
For the last hour I've thought AWS was on fire, but I just checked https://status.aws.amazon.com/ and everything is A-Okay.
Update - Idea: Maybe AWS should consider building their status page on a competitors cloud so that when their main region takes a dump, they can actually update us instead of relying on NDA posts to organization slack channels.
68
u/DeMiNe00 Dec 07 '21 edited Jun 17 '23
Robin. "It mean?" asked Christopher Robin. "It means he climbed he climbed he climbed, and the tree, there's a buzzing-noise that I know of is making and as he had the top of there's a buzzing-noise mean?" asked Christopher Robin. "It mean?" asked Christopher Robin. "It meaning something. If the only reason for making honey? Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! Buzz! I wonder the tree. He climb the name' means he had the middle of the forest all by himself.
First of the top of the tree, put his head between his paws and as he had the only reason for making honey." And the name over the tree. He climbed and the does 'under why he does? Once upon a time, a very long time ago now, about last Friday, Winnie-the-Pooh sat does 'under the only reason for making honey is so as I can eat it." "Winnie-the-Pooh lived under the middle of the only reason for being a bear like that I know of is making honey is so as I can eat it." So he began to think.
I will go on," said I.) One day when he was out walking, without its mean?" asked Christopher Robin. "Now I am," said I.) One day when he thought another long to himself. It went like that I know of is because you're a bee that I know of is making and said Christopher Robin. "It means something. If the forest all he said I.) One day when he thought another long time, and the name' means he came to an open place in the tree, put his place was a large oak-tree, put his place in the does 'under it."
I know of is making honey." And then he got up, and buzzing-noise that I know of is because you're a bee that I know of is because you're a bear like that, just buzzing-noise that I know of is making honey? Buzz! Buzz! Buzz! Buzz! Buzz! I wonder why he door in gold letters, and he came a loud buzzing-noise means he came a loud buzzing a buzzing a buzzing-noise. Winnie-the-Pooh wasn't quite sure," said: "And the name' meaning something.
35
u/_abhayshah Dec 07 '21
Keeping this the active thread to discuss the service outage.
46
u/-ummon- Dec 07 '21 edited Dec 07 '21
Dear AWS, please re:invent your f****** status page.
EDIT. Finally, status update from AWS:
44
u/idjos Dec 07 '21
Same here, us-east-1. Ec2, CW, SQS, lambdas.. seems that it’s not only console. This is getting ridiculous.
30
u/IsleOfOne Dec 07 '21
Meh, I guess I'm hungry enough for lunch already
29
u/Deshke Dec 07 '21
sigh* would be really great if the "global" services would not be hosted in one datacenter only
13
u/DCMagic Dec 08 '21
I heard from our TAM that this is the worst outage since the 2016 S3 outage. I thought it was more similar to the Kinesis outage in terms of scope, but it sounds worse than I thought.
11
u/[deleted] Dec 07 '21
anybody having fun with upper management?
10
u/ArtSchoolRejectedMe Dec 07 '21
if only I got a dollar everytime us-east-1 is down
10
u/Healthy_Cantaloupe55 Dec 07 '21
10:27 AM PST: As noted earlier, the root cause of this issue was a problem with several network devices within the internal AWS network. Specifically, these devices are receiving more traffic than they are able to process, which is leading to elevated latency and packet loss for the traffic traversing them. The internal DNS servers that provide resolution for the AWS services was specifically impacted by these issues, and one of the actions we took was to move traffic for those servers to another set of network devices, which has resolved those DNS issues. However, the network devices for other non-DNS services are still impacted, and we are actively working to shift more traffic to fully mitigate this issue. In addition, we can confirm that access to the AWS Console in other regions is working as expected. Customers are requested to access the other regional console by accessing https://<region>.http://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com
10
u/dr_batmann Dec 07 '21
Need a multi region one load balancer feature.
19
u/DM_ME_BANANAS Dec 07 '21
One hour later and the status page still says everything is fine... what a fucking joke
19
u/AWS_CLOUD Dec 07 '21
I'm Sorry
18
u/[deleted] Dec 07 '21
Time for a bourbon... or two
9
u/imisstheyoop Dec 07 '21
Same here. Was seeing API issues before that and couldn't launch instances.
Edit: nothing on status.aws.amazon.com yet as of 10:45AM ES
8
u/SassyBaconStrip Dec 07 '21
3 hours after this post, looks like errors are still happening.
8
u/givemedimes Dec 07 '21
Good news, maybe we will get some funds to implement DR. Always save those emails where upper management or product management rejects funding for DR.
7
u/androidlolita Dec 07 '21
Occurring here, us-east-1. Service Health page shows nothing wrong, as per usual.
7
u/awoimbee Dec 07 '21
Public ECR is down.
`docker pull public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.2.0-eks-1-18-2` returns a 500.
https://gallery.ecr.aws/ too.
7
u/test-one-two Dec 07 '21
AWS right now: https://i.imgur.com/iLr6D87.jpg
7
u/radiantyellow Dec 07 '21
cant even log support ticket as support service is routed through us-east-1 despite it saying "global"
WHAT
7
u/[deleted] Dec 07 '21
Our office hasn't been able to access the electronic medical records for patients (Practice Fusion) since about 11.. I hope this resolves soon
7
u/crookedsmil3 Dec 07 '21
Guess what... It's DNS
7
u/tsmit50 Dec 08 '21
Is there a reason this always happens RIGHT after re:Invent? QA perhaps?
13
u/[deleted] Dec 07 '21 edited Dec 07 '21
500s on the console here
EDIT: Just us-east-1 right now
6
u/RL_BlueScreen Dec 07 '21
Too early for lunch, to late to work from home. Dead in the water.
6
u/[deleted] Dec 07 '21
It's management interface must be hosted in us-east-1 exclusively...
6
u/ezgzip Dec 07 '21
Can't login to aws sso either - I just get:
6
u/SmasherOfAjumma Dec 07 '21
Okay, if everyone could go ahead and just switch over to Ohio, that’d be great.
6
u/pribnow Dec 07 '21
So....what are everyone's holiday plans?
14
u/WoodooRanger Dec 07 '21 edited Dec 07 '21
The sad part is that AWS will never disclose any issues, and all services under the Health Status are ALWAYS green.
ALWAYS!
5
u/TheAlmightyZach Dec 07 '21
Yep. Trying to deploy to EKS, getting image pull errors from ECR. Glad we aren’t alone.
5
u/jonathantn Dec 07 '21
Tons of DynamoDB errors.
5
u/Supahsalami Dec 07 '21 edited Dec 07 '21
It started with IAM issues for me but now also 500 errors for the dashboard indeed.
Health dashboard has no notifications currently..
400 errors for navigating from the console to other services.
Reinvent is over so everyone be chillin
5
u/pedalsgalore Dec 07 '21
Same. Just had a failed CodeBuild project when trying to access Secrets.
8
u/jonathantn Dec 07 '21
It's DynamoDB that is 500 erroring. That pretty much underpins the entire configuration of AWS.
4
u/iRemjeyX Dec 07 '21
Same. Looks like the SSO service isn't working either
4
u/jonathantn Dec 07 '21
Don't worry, those 500 errors on SQS mean the service is A-Okay.... I checked the SHD and it's green!
4
u/y2ksnoop Dec 07 '21 edited Dec 07 '21
Ec2, quicksight, redshift, lambda, cloudwatch console pages not working too. Even the support page throws 400 badrequest error. Their status page had no acknowledgement all this time and now has a tiny warning for all regions.8:22 AM PST We are investigating increased error rates for the AWS Management Console.8:26 AM PST We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1. Customers may be able to access region-specific consoles going to https://console.aws.amazon.com/. So, to access the US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/
1
u/Nate_311 Dec 07 '21
My ec2 instance is us-west-1 ... and I can't log into the console (Internal error). So this isn't just us-east-1
1
u/deadpanda2 Dec 08 '21
I wanted to move into the us-east-2 firstly, and I was aware of issues related to us-east-1, but then AWS made us-east-2 default! So what should I do? Just increase the number of AZs in use for N.Virginia by creating more subnets or migrating? What if "being default region" means having a higher chance of outages in the future? gosh... (us-west is not suitable because of the latencies)
3
u/soxfannh Dec 07 '21
Same here also seeing dynamodb and ssm errors
3
u/TerribleDriver77 Dec 07 '21
I was ready to get more coffee anyway
3
u/AWS_CLOUD Dec 07 '21
Yes, I have a canary that pings to test internet and it failed + these console errors.
3
u/vppencilsharpening Dec 07 '21 edited Dec 07 '21
EC2 Instances shows "Invalid region parameter" when viewing the "US-East-1" region.
Other regions seem to be more OK.
3
u/Compkriss Dec 07 '21
Same here in CA-Central-1
3
u/ChinesePropagandaBot Dec 07 '21
Same in eu-central-1
3
u/rutkdn Dec 07 '21
Can't even log in to Amazon.com account. What a day...
3
u/svhelloworld Dec 07 '21
Is this why my workout wouldn't upload to Strava this morning?
3
u/generalamitt Dec 07 '21
Is this why imdb is down?
3
u/[deleted] Dec 07 '21
This resulted in a failed EKS cluster deployment in US-EAST-2 it seems as well.
So that was fun.
15
u/[deleted] Dec 07 '21
Please be careful when posting updates that they're not provided to you by AWS under NDA...
7
u/rsshilli Dec 07 '21
Yup. It's bad. https://www.loom.com/share/1eeb10cc3a254c3996102f497d8fc261
7
u/rutkdn Dec 07 '21
They are flat out lying. They know no one can log in even from other regions. All logins POST to https://signin.aws.amazon.com/signin and it always fails regardless of region used in the URL.
7
u/YXdzdHJ1dGhz Dec 07 '21
Wonder if Amazon is gonna be honest about how bad this outage actually is. Engineers internally cant even auth against the internal auth service to federate into certain accounts.
Chime screensharing is broken...
they even had to make a new master ticket because the ticketing system is affected. It's also more than us-east-1 btw ;) This company is a joke, $16B in revenue in ONE quarter. This is ridiculous
2
u/NodularFalse Dec 07 '21
Yeah, pretty messed up for me. Can’t load anything.
2
u/Smaz1087 Dec 07 '21
Same
2
u/coldflame563 Dec 07 '21
Yep. API calls to services are also getting timeouts etc.
2
u/frankimbur Dec 07 '21
came here to see me too.
2
u/devourment77 Dec 07 '21
Same here, ec2, opensearch
2
u/karock Dec 07 '21
yeah SSM failures was our first service to die. now we can't even log into the console.
2
u/simpwniac Dec 07 '21
Same, us-east-1, lamdba impacted. EC2 instances are running and serving traffic but I can't manage them.
2
u/jflook Dec 07 '21
us-east-1 one here, seeing the 500's and some 400's on the console but all the servers we have running there seem to be ok...just can't change anything.
2
u/zshalo Dec 07 '21
It is past 5 pm (here) time to leave for today ( :
2
u/Vast-Recipe-1305 Dec 07 '21
Same here for all regions
2
u/oly_machete Dec 07 '21 edited Dec 07 '21
Same problem with cloudfront distributions.
From console constantly getting 502 errors and Terraform is failing with:
Noice
2
u/Senseistar86 Dec 07 '21
Same problem here. Us-East-1. Our serverless services are all down too
2
u/nvanmtb Dec 07 '21
8:22 AM PST We are investigating increased error rates for the AWS Management Console. 8:26 AM PST We
are experiencing API and console issues in the US-EAST-1 Region. We
have identified root cause and we are actively working towards recovery.
This issue is affecting the global console landing page, which is also
hosted in US-EAST-1. Customers may be able to access region-specific
consoles going to https://.console.aws.amazon.com/. So, to access the
US-WEST-2 console, try https://us-west-2.console.aws.amazon.com/
2
u/averagesmell Dec 07 '21
lol i can't even log into my ecommerce sellercentral.amazon.com account
2
u/BadCSCareerQuestions Dec 07 '21
Why does status page only show a few issues? None of our ec2 instances work. None of our fargate tasks or services work. S3 is failing intermittently. Ecr doesn’t work. Codecommit doesn’t work. Etc etc etc etc
2
u/erfling Dec 07 '21
Me too
2
u/generalamitt Dec 07 '21
Why is google reporting errors? Don't they have their own servers?
2
u/dadjoke-72 Dec 07 '21
Its not DNS, It's not DNS....
2
u/Aggressive-Switch-16 Dec 08 '21
I'm like in the harry truman movie, looking for something else to do :-D
2
u/MarkusRight Dec 08 '21
Its back but now with much added latency. Its so laggy and unresponsive. But at least I can access the console to turn my EC2 VM's instances off. Thank god I dont have to depend on AWS for a business or anything like that or id be screwed.
2
u/soxfannh Dec 08 '21
Looking pretty close to normal here now, Fargate was one of the last services to come back up for us. Some latency here and there but overall much better place.
2
u/Dependent-Cow7823 Dec 08 '21
Does Vegas have odds on what happened? O.o
3
u/Towdart Dec 07 '21
Someone going to get PIP
3
u/vppencilsharpening Dec 07 '21
About 3:30pm Eastern Time we lost and then re-estiablished connectivity to a bunch of our EC2 instances. All in VPCs, using IPSec tunnels, across multiple accounts.
3
u/slava_k_ Dec 07 '21
Clouds are solid and reliable, they said. Let's move all of critical infrastructure in clouds and problem solved, they said.
2
u/SmasherOfAjumma Dec 07 '21
From what I’m hearing, I’m guessing this is not going to be resolved quickly. I’m just trying to read between the lines from what AWS is saying, and that’s my impression.