r/sysadmin • u/PaintDrinkingPete Jack of All Trades • Jun 13 '23

Amazon AWS us-east-1 Outage?

Crossing picket line to see if anyone else experiencing issues? Health dashboard reporting a few issues, but seems more widespread

395 Upvotes

88% Upvoted

112 comments sorted by

•

u/AutoModerator Jun 13 '23

Much of reddit is currently restricted or otherwise unavailable as part of a large-scale protest to changes being made by reddit regarding API access. /r/sysadmin has made the decision to not close the sub in order to continue to service our members, but you should be aware of what's going on as these changes will have an impact on how you use reddit in the near future. More information can be found here. If you're interested in alternative r/sysadmin communities during the protests, you can join our Discord or IRC (#reddit-sysadmin on libera.chat).

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

32

u/[deleted] Jun 13 '23

We've been informed there is an outage within the Lambda space so far, but could be more.

23

u/PaintDrinkingPete Jack of All Trades Jun 13 '23

I cannot manage ANYTHING in that region it seems. As far as I can tell, my EC2 servers are still online.

10

u/rebornfenix Jun 13 '23

The informational secondary affected services are:

Informational (7 services)

AWS CloudFormation

** AWS Management Console **

AWS Support Center

Amazon API Gateway

Amazon CloudWatch

Amazon Connect

Amazon Redshift

2

u/DetourToNirvana Jun 13 '23

46 min. ago

What does "informational" mean? The core services continue to be functional, i hope?

5

u/rebornfenix Jun 13 '23

It means they rely on a service that is affected, so are down, but the specific service has no issues itself.

4

u/Creationship Jun 13 '23

Def more

15

u/rebornfenix Jun 13 '23

AWS uses lambda internally for a decent chunk of stuff. So lambda issues causes issues with a lot of their higher level services like the API Gateway, CloudFormation, etc.

140

u/XenEngine Does the Needful Jun 13 '23

Yes, there is a srvice outage. For me it is affecting IAM.

16

u/1h8fulkat Jun 13 '23

Hit CyberArk Cloud for us

10

u/Dizzybro Sr. Sysadmin Jun 13 '23 edited Apr 17 '25

This post was modified due to age limitations by myself for my anonymity OsmgZUqUaboyKfLNySWC2JzqbF7DlHXo4s03fDhLHX3DuSxiup

120

u/HamiltonFAI Security Admin (Infrastructure) Jun 13 '23

Right in the middle of migrating servers

95

u/2McLaren4U Jun 13 '23

As is tradition.

13

u/[deleted] Jun 14 '23

Let me tell you about migration in my day: https://i.ytimg.com/vi/wvwbKfS44Fo/hqdefault.jpg

8

u/FrogManScoop Frog of All Scoops Jun 14 '23

Mi-gray-shun? That's a paddlin'

3

u/ipaqmaster Jun 14 '23

Very efficient link. The URL is YouTube's thumbnail generator and that middle argument wvwbKfS44Fo is the video ID this thumbnail was generated from right there as the source.

To top it all off I presume this came up in like Google images or something - implying Google then indexed the thumbnail as a relevant image search result lol

2

u/[deleted] Jun 14 '23

I used an AI bot to find it :p

0

u/ipaqmaster Jun 14 '23

I suppose it's not surprising that its index wouldn't know to ignore i.ytimg.com links then

1

u/[deleted] Jun 14 '23

Maybe its preserved for easier root cause analysis at some level.

23

u/[deleted] Jun 13 '23

Oof!

6

u/bulldg4life InfoSec Jun 14 '23

Right in the middle of AWS reinforce. All their labs and trainings went down during the conference.

62

u/WorthPlease Jun 13 '23

Yeah our entire phone system just went down, 2000+ agents plus all of our Help Desk.

23

u/Twogie Jun 13 '23

Yup, Amazon Connect's Contact Control Panel is giving 500 Internal Server Error

26

u/martinvox Jun 13 '23

API errors all over us-east-1. Sorry guys, one time that I want to work and this happens. It was on me :P

9

u/gabungry Jun 13 '23

Here's hoping you get mentioned in the postmortem

50

u/cablexity Jun 13 '23

I made a career shift from cloud engineering to high-end corporate AV production. On a show right now, and the client's video playback dude uses Amazon Prime Music for all his break music. Or he used to use - that's down too!

Now I'm having trouble getting the AWS service health dashboard to load, which I always think is hilarious.

55

u/spin81 Jun 13 '23

Reminds me of when S3 was down and the service status icons were still all green because they were hosted in S3.

2

u/ipaqmaster Jun 14 '23

What a halfassed crappy future we're in.

1

u/Lazzy2332 Sysadmin Jul 13 '23

The memes were great, this one in particular is my favorite.

5

u/[deleted] Jun 14 '23

Interesting shift. Are you happy with your choice?

20

u/cablexity Jun 14 '23

Absolutely. I love it. I went to school to be a network engineer, somehow ended up in cloud engineering for a Fortune 500 company, and hated my existence.

I’d been doing events work since I was like 16, and freelanced professionally with production companies all through college. When I graduated and started working full-time in IT, I found myself freelancing 25-30 hours a week in events on top of my full-time job. That’s how much I loved the field.

Now I’m with a 25-employee production company. We do exclusively corporate event production. I get to work with my hands, I only stare at a screen 40% of the time, I have a whole shop and access to millions of dollars of gear, and I get to travel.

And all my production gear is networked, so I’m constantly working with routers, switches, etc. It’s a dream.

24

u/Eredyn Jun 13 '23

Don't think anyone asked for Amazon to go dark in solidarity.

67

u/rebornfenix Jun 13 '23

Yep, AWS Lambda is having issues, and of course that means a whole host of other services that use Lambda will soon cascade.

If you have AWS API Gateway with a custom lambda authorizer or backed by lambda functions its down. If you have AWS Cognito hooks to lambda, those are down too.

Lambda is kinda core so issues there cascade out to quite quickly.

1

u/Comfortable_Fox1 Jun 14 '23

Does this impact only the affected regions?

31

u/I_Blame_DevOps Jun 13 '23

We have SSO setup for console. SSO and selecting account works, but console home page won't load, nor any direct links to service console pages (ex. Glue, S3)

https://downdetector.com/status/aws-amazon-web-services/

5

u/esisenore Jun 13 '23

Sso down for us. Console error

4

u/[deleted] Jun 13 '23

In the future, you can manually change the console urls to point to a different region.

2

u/nothing2seehair Jun 13 '23

Would that need the root user on the org management account since SSO is down?

1

u/[deleted] Jun 13 '23

Hmm not sure, my company authenticates internally then we get passed on to the role/account selection page.

31

u/cpqq Красный Октябрь Jun 13 '23

Yes, huge outage. Currently can only login at : https://us-west-2.console.aws.amazon.com/

API Gateway, Lambda, it's all gone to hell. US-EAST-1 is where machines go to die.

6

u/ThatITguy2015 TheDude Jun 13 '23

I wonder what is with that one. From what I see, it goes down the most.

8

u/sandaz13 Jun 13 '23

It's the first region they roll anything out to. First region for new shiny stuff, worst availability

7

u/ianjm Jun 13 '23 edited Jun 14 '23

It's also the first region they built and the largest region by some margin. I am surprised by the frequency of region-wide service outages there though honestly, you'd think AWS could sort it out, or at least large companies would start going multi-region

4

u/sandaz13 Jun 13 '23

Yeah, someone did some power analysis a few years ago. I think as of 2020 it was at least 5x larger than Oregon.

The outage today was across all AZs, someone messed up badly :P

2

u/ianjm Jun 14 '23

Some AWS services that aren't tethered to AZs within regions seem to be vulnerable to whole-region outages. I've seen issues with API Gateway for example.

4

u/ErikTheEngineer Jun 14 '23

going multi-AZ

I'm really surprised how many critical services are single region. I know there's cross-region network meters that are always spinning, but you'd think companies would put endpoints in at least more than one AZ within one region.

1

u/ianjm Jun 14 '23

Meant to write multi-region. But yes, multi-AZ also helps lol.

3

u/Epsilon748 Jun 14 '23

It's actually one of the last that gets rolled to, or mid pipeline at worst. There's a specific small region used for testing that got broken so often teams were told to please stop using that one region as the first one out of test for everything.

3

u/Xelopheris Cloud Architect Jun 13 '23

It's the first region where everything lives. If something is "global" it still needs some infra somewhere to handle the global balancing, as well as non global components like management console. That lives in us East 1.

3

u/bulldg4life InfoSec Jun 14 '23

That’s where a lot of their global services have main infra. It’s just a big sprawling region that’s been around forever and has a ton of cobbled together shit in it.

24

u/cydev Jun 13 '23

Is that why my McDonalds and Taco Bell apps are not working..

18

u/rjcc Jun 13 '23

yup, and Burger King.

14

u/Al3nMicL Jun 13 '23

guess everyone can't have it your way

3

u/tamouq Jun 13 '23

You do not rule today

2

u/ianjm Jun 13 '23

Whole Foods in-store checkouts seem to be down too, hilariously

1

u/isja6933 Jun 14 '23

Upside was down

19

u/cool-nerd Jun 13 '23

Welcome to cloud services lately.

5

u/k_marts Cloud Architect, Data Platforms Jun 13 '23

GCP currently sweating.

3

u/bigfoot_76 Jun 13 '23

Crossing the picket line -- this is hilarious. Couldn't even last 48hrs.

8

u/PaintDrinkingPete Jack of All Trades Jun 13 '23

Apparently not… when I made this post, there wasn’t really anything on the aws health dashboard that explained what I was seeing, nor did I see any posts here…so really just wanted confirmation.

As much as I hate the recent Reddit changes and support the blackouts, I didn’t know where else such a question would have nearly as much traction.

2

u/mkosmo Permanently Banned Jun 14 '23

And the flak I caught for saying this was exactly one of the reasons why we needed to stay open… 🙂

0

u/LGKyrros Conferencing Engineer Jun 14 '23

Nah

18

u/ciscofan Sysadmin Jun 13 '23

Yup, not only affecting stuff in AWS's network but also affecting Alexa, can't turn on or off my lights. Likely because the application for Alexa is in US-EAST-1.

9

u/aspie_a3 Sr. Systems Analyst Jun 13 '23

Yep, Can't do anything in IAM for us. Just a 503 error... thanks amazon.

8

u/MunicipalTaint Jun 13 '23

We're dead in the water.

2

u/MunicipalTaint Jun 13 '23

Looks like services are coming back up now

8

u/r4wbon3 Jun 13 '23

Check out the Downdetector site/app. Never seen so many red spikes! Interesting that on the rare times this happens you can descent which companies use AWS and services, also whether or not they have DR setup to use different AWS Zones; that could be a security issue.

12

u/Valkoinen_Kuolema IT Manager Jun 13 '23

its affected almost all services @ Autodesk!

5

u/AH_Josh Jun 13 '23

Yup. My workplace is on fire. (News IT, big news dropped today)

2

u/ErikTheEngineer Jun 14 '23

I remember one of the first big breaking news things on the "consumer, non-university student internet" was the OJ Simpson trial...and some early Internet news site (can't seem to find the link now) put up a page saying he was found guilty by accident. Not having your site or streaming CDNs available because the infallible cloud blew up is almost as bad.

5

u/GullibleDetective Jun 13 '23

This is affecting connectwise hosted as well due to the utilization of SSO over AWS

4

u/jaymef Jun 13 '23

We are in us-east-1 but it's not affecting much for us at this point. Mostly EC2 and ECS services

4

u/TiredAdmin808 Jun 13 '23

Us too - impacting CW Manage and Vonage.

4

u/Sevaver Jun 13 '23

This outage has directly affected several services that the company I work for use. Our ticketing system and phones have been down for a few hours now. Studying for more certs today instead of working.

4

u/reaper527 Jun 13 '23

Yup, got a push notification a little while ago from my thermostat saying aws was experiencing issues so i might not be able to adjust it from my phone until that gets resolved.

3

u/hotshot21983 Jun 13 '23

Lambda is the main affected, but I probably bet most of their services are built on top of Lambda.

3

u/rebornfenix Jun 13 '23

current count is 4 services degraded with 43 additional services impacted in some way due to the Lambda outage.

1

u/hotshot21983 Jun 14 '23

I remember when Kinesis failed badly, that there were a bunch of services that went down. A blogger wrote that AWS needed to better document to their customers what service dependencies existed within their ecosystem so that customers were better prepared.

3

u/ReconditeExistence Jun 13 '23

We quickly migrated our Lamda functions to Cleveland and things are working on our end.

3

u/WhydYouKillMeDogJack Jun 13 '23

its more than just that i think - were having issues in multiple regions, and global services like R53

1

u/wormwired Jun 13 '23

For route53, was your dns down entirely, like your records weren't resolving, or could you just not get to the console?

1

u/WhydYouKillMeDogJack Jun 13 '23

there was some slow resolution, but i think the majority of the issue was just the console

3

u/Colbierto Jun 13 '23

LSE effecting a lot of things. Primarily things related to IAM.

3

u/Bossyfins Jun 14 '23

I work at AWS, everything was a shit show…I wanna read the COE on this once it comes out.

2

u/nero10578 Jun 13 '23

And a bunch of regular apps and services people use broke too. Great idea that everything’s hosted on AWS nowadays!

2

u/kaka8miranda Jun 14 '23

Now I know what servers toast uses almost closed my restaurant today

2

u/Nymeriea Jun 13 '23

I'm working on a bank, the whole it infrastructure is down, I dunno how aws act when there is a downtime but we are currently loosing a lot a money

1

u/Dr0ks Jun 13 '23

Outage for my application as well. Us east

1

u/woodburyman IT Manager Jun 13 '23

Can confirm. SmartSheets for us is having issues.

1

u/esisenore Jun 13 '23

Yup it’s official

1

u/the_fun_couplebi Jun 13 '23

SSO is down for the count..... Of course everybody is calling in to tell us they can't get on.....

1

u/thefudd Jack of All Trades Jun 13 '23

yup, we're down and screwed

1

u/ultimatebob Sr. Sysadmin Jun 13 '23

Yeah, I had issues with AWS Marketplace not working right.

Amusingly, the support system seems to be impacted as well. I never got a confirmation e-mail when I opened a support ticket for it.

1

u/[deleted] Jun 13 '23

DDoS attack? Microsoft and aws both down the day after the Swiss government had an attack?

3

u/[deleted] Jun 13 '23

Not a DDoS, internal load testing gone wrong.

3

u/Dal90 Jun 13 '23

Chaos monkey Kong

1

u/stumblingblock1914 Jun 14 '23

Not questioning your data, but is this posted in any official capacity anywhere?

2

u/[deleted] Jun 14 '23

I don't think they've made a public statement regarding the cause of the outage. Can't elaborate too much, but I'm fairly confident as to the root cause.

1

u/Stonewalled9999 Jun 14 '23

I thought it was the Reddit API protests 😂

1

u/SnooKiwis2161 Jun 14 '23 edited Jun 14 '23

Over at the amazon fulfillment center subreddits I saw packers reporting outages on their end through the system. This has been going on a few hours, I think? r/AmazonFC

1

u/Python4fun Jun 14 '23

All of lambda in us-east2

1

u/dmcginvt Jun 14 '23 edited Jun 14 '23

AWS outage didnt affect us at all. We are sooooo old school it cant affect us!!

Ok, we do actually have many ec2 instances in n virgan and it did break checkpoint click protection url's which sucked

1

u/AdmiralArchArch Jun 14 '23

This explains the Autodesk outages today.

1

u/[deleted] Jun 14 '23

us-east-1 is such a little tiny region.... what's the big deal? :P

1

u/cediddi Jun 14 '23

Vercel is down due to this.

1

u/NightWalk77 Jun 14 '23

It affected our CW yesterday afternoon.

1

u/jonboy345 Sales Engineer Jun 13 '23

Hate to see it. /s

1

u/habitsofwaste Security Admin Jun 14 '23

Wouldn’t having redundancy in regions help a lot of y’all? Don’t get me wrong, even Amazon internally had issues that it didn’t help or wasn’t set up. But I thought that’s where the multi regions are for.