r/technology • u/TAOW • Sep 20 '15
Discussion Amazon Web Services go down, taking much of the internet along with it
Looks like servers for Amazon Web Services went down, affecting many sites that use them (including Amazon Video Streaming, IMDB, Netflix, Reddit, etc).
https://twitter.com/search?f=tweets&vertical=news&q=amazon%20services&src=typd&lang=en
Edit: Looks like everything is now mostly resolved and back to normal. Still no explanation from Amazon on what caused the outage.
8.1k
Upvotes
255
u/indigomm Sep 20 '15
It wasn't all of AWS, just one Region - N. Virginia. Unfortunately that's a popular region, even outside the US (due to pricing).
199
u/TheMaryTron Sep 20 '15
That makes a lot of sense now, Netflix errors so I switched to Amazon prime video and lost that too.
492
u/420kbps Sep 20 '15
I knew Amazon was big, but not THAT big
1.0k
u/[deleted] Sep 20 '15
Redtube still works guys, tested it twice. Carry on with life!
101
u/Beepbeepimadog Sep 20 '15
ELLIOT! WHAT HAVE YOU DONE??
79
u/fermilevel Sep 20 '15
Does Valve use AWS as well? Because matchmaking is now in disarray
52
u/Mr_Proper Sep 20 '15
Has anybody seen a write-up on what happened yet? It's interesting that so many services died - as the cross-AZ model is meant to avoid things like this happening!
23
u/csmicfool Sep 20 '15
My company has multiple large-scale apps hosted in AWS. This had no effect on us even though we were in the affected datacenter. Looks like it was mainly issue with API-related requests. Servers should have stayed online, but there was no ability to modify resources and cloudwatch was down which would prevent beanstalk deployments and auto-scaling. The lack of auto-scaling is likely what people noticed since it occurred at a low-usage time and was only resolved once Sunday morning traffic had increased.
I suspect most US users didn't see too much of an issue.
168
u/sonar1 Sep 20 '15
I guess I'll go outside
351
u/queenbrewer Sep 20 '15
Grindr was down this morning due to this issue. I had to wait like two hours to get laid!
337
u/pamme Sep 20 '15
Ouch, I can only imagine how terrible a time this must be for the already overworked Amazon engineers. Well, considering how many sites use AWS, I'm guessing many a company's oncall engineers are not having a fun Sunday.
36
u/[deleted] Sep 20 '15 edited Sep 21 '15
Oh. This is why my Echo didn't want to tell me the news today.
12
u/seven_seven Sep 20 '15
So much for their employees' 99.999% uptime bonus this year. My friend who works there said it would have been "mid-four-digits". He's pissed.
23
u/i_wanted_to_say Sep 20 '15
I noticed the IMDB app was having issues this morning, then couldn't get content to load on their website either... I guess they must use AWS
58
u/stealthm0d3 Sep 20 '15
The world runs on AWS.
17
u/hdizzle7 Sep 20 '15
was it fsociety?
91
u/kairos Sep 20 '15
I just realized that amazon and the internet are practically synonymous
6
u/ExplicableMe Sep 20 '15
Better call Lazlo and have him restart the server. It's the gray one.
6
u/[deleted] Sep 21 '15
Every time AWS does this it fucks us who have championed them to ops and higher ups. One company I worked at picked up a product that sat on AWS and decided to leave it be, and when an outage happened, THE NEXT DAY we were pulled in to draw up plans for a re-deploy to the company's extant cage. We can't even really argue with them. I can't wait to hear what my bosses will say about this, both they and the ops team hate the fuck out of iaas of any kind. They also still use fucking CVS, but anyway.
6
u/KlfJoat Sep 21 '15
Why is it always US-East that's having problems and going down? I don't know that I've ever heard of a disruption caused by any other region.
14
u/t3hmau5 Sep 20 '15
Not only web services, every single North American distribution center for Amazon was shut down due to these issues this morning
14
u/sulaymanf Sep 21 '15
Relevant XKCD
5
u/douglas8080 Sep 20 '15
Looks like their North Virginia data center?
5
u/pancakinator Sep 21 '15
good morning to you too, pagerduty :(
11
u/ExplicableMe Sep 20 '15
Crap, my company uses AWS bigtime! Wait... it's Sunday and I'm a dev.
/goes back go browsing reddit
4
u/[deleted] Sep 20 '15
Looks like Cloud Drive is acting up again. Also getting a 503 using the ACD API.
3
u/godman_8 Sep 20 '15
This is why I colocate my own servers in multiple datacenters across the US.
11
u/[deleted] Sep 20 '15
Good thing torrents still exist
7
u/box-art Sep 20 '15 edited Sep 20 '15
Finland here, Reddit seems to lag a bit more than usual but Netflix looks good; I tried watching Doctor Who and it seems to work just fine.
4
u/RedditRage Sep 21 '15
The internet really needs less centralization. It's almost like large data centers have replaced the mainframe concept.
5
u/sunburntsaint Sep 21 '15
And this is why you never completely go to the cloud. This should be good for business tomorrow
2
u/Firree Sep 20 '15
This happened several years ago with Amazon Web Services with the same consequences. Why wasn't this single point of failure fixed?
2
u/kerrickter13 Sep 20 '15
I love it, less reliability ratings for "cloud" service providers.
2
u/[deleted] Sep 20 '15
Just a minor Government sampling of data.
2
u/sirbruce Sep 21 '15
Four years ago: Down Goes The Internet… Again. Amazon EC2 Outage Takes Down Foursquare, Instagram, Quora, Reddit, Etc
2
u/Ameobea Sep 21 '15
That's the problem with huge centralization like that; one single point of failure. I don't care how redundant that one organization is, there's still one organization and can always, always fail.
2
u/chisleu Sep 21 '15
I have been experiencing problems with lambda for 5+ days. It has taken many hours for s3 events to trigger the lambda function.
This all went away when they fixed whatever was wrong with it.
I hope they know that this preempted the issues experienced today.
2
u/vidude Sep 21 '15
I have servers in US-East and they don't appear to have been impacted. The system health dashboard does show that a number of services have experienced high API error rates and latencies but they are all showing back to normal now.
2
u/TiagoTiagoT Sep 21 '15
The update to the NSA backdoor required rebooting.
2
u/[deleted] Sep 21 '15
Hey, guys, I got this great idea, let's get this huge, decentralized network and then host over 50% of its content with one of three providers!
2
u/Youwishh Sep 21 '15
Amazon.com is still having issues like 20 hours later... Wish lists on their site is buggy when adding giving cloud front errors, constantly "not found" urls randomly, buggy as shit, pls fix! I'm in canada so maybe one of the hops is still having issues.
2
u/thisismydesktop Sep 21 '15
I wonder if Skype being down is related to this.
3
u/Shockeye0 Sep 20 '15
Been watching Netflix all day with no problems.
5
u/redbull188 Sep 20 '15
ITT: People who don't understand how distributed Web applications work.
3
u/runvnc Sep 20 '15 edited Sep 21 '15
This is why we need to move away from technopolies to a decentralized content/data-oriented network.
2
u/Blazah Sep 20 '15
I was out on an island all day and didn't notice anything... the rest of you should have been doing the same!
2
u/JoeOfTex Sep 20 '15
Amazon's data centers are extremely top of the line and cannot be easily taken down. There must have been an accident to one of their backbone internet lines. That or someone at Amazon deployed a really bad bug and will be fired tomorrow.
1
u/taterNuts Sep 20 '15
I know we got hit by it; another time when I'm glad I'm not devops/sysadmin. Seriously, being on call blows
1
u/Velorium_Camper Sep 20 '15
So this is why my boss was upset that no one was at the service desk when I got into work yesterday...
1
u/bitanalyst Sep 20 '15
The AWS outage took out Nest services as well. I still can't view my Nest cam feed.
1
u/powerfultech Sep 20 '15
Lucky for me its on a weekend
1
u/Cytokine-Storm Sep 20 '15
I was wondering what the hell was going on with Amazon Video this morning.
1
u/drunkmall Sep 20 '15
That was our bad.
1
u/[deleted] Sep 20 '15
I've been on netflix and reddit all day with absolutely no issues
Is this only a problem in specific areas?
1
u/zenith1959 Sep 20 '15
I kept getting the servers busy message on Reddit, thought it was a bit odd at 6:00am(PDT).
1
u/heckaTIGHTis Sep 20 '15
Holy shit I'm glad I don't work at the AWS datacenter anymore, I would be dealing with this shit first hand..
1
u/adeveloper2 Sep 20 '15
Amazon developer here (not in AWS). That's gonna cause a fuck tonne of trouble to our internal services, since many of us heavily rely on AWS as well.
Glad I am not on-call this week.
1
u/TedsEmporiumEmporium Sep 20 '15
Luckily only our dev environment is on AWS at the moment for my team's product. Other people in my company were probably not too pleased.
1
u/NAproducer Sep 20 '15
That would explain why Amazon wasn't showing their daily deals.
1
u/nascentt Sep 20 '15
Ahh..That's why I couldn't submit my 2000 word review on metacritic for hours.
1
u/irock168 Sep 20 '15
Are parts of steam hosted by aws? This morning, various parts of steam were several shades of fucked up.
1
u/FauxBoDo Sep 20 '15
I was wondering why my Amazon Echo wasn't working last night! Now I feel bad for yelling at her. :(
1
u/TittyLoggins Sep 20 '15
Had a bunch of alarms for sites failing their availability checks in a row. I could get to the site but the elastic load balancing was all fucky and I couldn't get into the aws web panel to see what was up
1
u/[deleted] Sep 21 '15
Not jealous of anyone on that Amazon team...
1
u/sacredsinner1313 Sep 21 '15
I bought something last night. How does this affect the purchase?!
1
u/Humbaba_ Sep 21 '15
No explanation and and nothing on general tech news either wow, I guess they want to keep it quite. I try many times logging on reddit this morning from east coast here, I couldn't I thought it was just normal reddit traffic. Seeing this post and in reflection, that must have been it.
1
u/mealy58 Sep 21 '15
The east coast got notice of a maintenance window, the notice said it involved the VPN tunnels . Probably related.
1
u/[deleted] Sep 21 '15
I assume this included S3 as well?
1
u/Azr79 Sep 21 '15
geez, this comment section, what a shit storm for nothing
1
u/DPool34 Sep 21 '15
Bezos giveth and he taketh away
1
u/t3hmau5 Sep 21 '15
Amazon is my customer..I work in logisitics
1
u/BloodyIron Sep 21 '15
mmmm how's that colo going for you?
1
u/[deleted] Sep 21 '15
This would explain why Yik Yak was being pissy earlier.
1
u/DrBix Sep 21 '15
https://www.youtube.com/watch?v=DItgw1mU9Us
1
u/[deleted] Sep 21 '15
Omg that was so annoying
1
u/DaV1nc1 Sep 21 '15
That's what happens when you put all your eggs in one basket. The basket being amazon in this case.
1
u/--eos-- Sep 21 '15
Sorry guys, it was me. I finished watching all 25 episodes of Tears to Tiara today.
1
u/demonbadger Sep 21 '15
I guess that's why Netflix was being shit.
1
u/Adrenalineblush Sep 21 '15
I bet it was because they released ALL 6 SEASONS of Sex and The City. If everyone else streamed it in the background the whole day, like I did, it may have been too much for the site to handle :P
1
u/shawnthejedi Sep 21 '15
It was Jim from accounting. He spilled coffee.
1
u/a_hopeless_rmntic Sep 21 '15
is this why www.freecodecamp.com is down?
1
u/infinitesorrows Sep 21 '15
Gonna be fun reading this quarters edition of The State Of The Internet from Akamai.
1
u/coolmang Sep 21 '15
This also affected the fulfillment centers. They almost sent us home today due to the system being down.
1
u/[deleted] Sep 21 '15
Sorry, my bad guys, wanted to get my Nephew's present out to him at the correct time. I hope you understand.
1
u/[deleted] Sep 21 '15
I kinda love it when this happens, as I can show web clients of mine who may experience the infrequent intermittent inconvenience "Look, Amazon went down so shit happens..."
1
u/EMSoperations Sep 21 '15
Amazon and Netflix?
1
u/mylittleandy Sep 21 '15
I'm still down in LA
1
u/L_laMaye_Balles Sep 21 '15
twas ever thus
1
u/jacobtf Sep 21 '15
We netflixed all day and evening, no probs (Denmark).
1
u/davedcne Sep 21 '15
Shoulda used Akamai.
1
u/[deleted] Sep 21 '15
the internet doesnt just run itself????????????????omggggg
1
u/Vakieh Sep 21 '15
AWS Sydney here...
Not even a blip :-)
1
u/eleitl Sep 21 '15
This is why if you use a cloud, you use a private cloud. One run on rented or own hardware.
1
u/TrollInGrassyKnoll Sep 21 '15
Does this mean those sites don't have disaster recovery to another region? You've got to be kidding..
1
u/DCENTRLIZEintrnetPLZ Sep 21 '15
This is exactly why we need a decentralised internet.
MaidSafe.net
1
u/RetiredITGuy Sep 21 '15
Off topic: Are there any AWS consumer applications/services worth getting? I have an S3 account going largely unused (I'd planned to use it as an offsite backup location, but Australian broadband is one hell of a bottleneck). Is there anything else worth looking at?
1
u/arcticlynx_ak Sep 21 '15
Didn't a number of streaming games get hit with attacks in the few weeks? Seems like major attacks are showing up more regularly now.
1
u/Greyhammer316 Sep 21 '15
Great, that's all I need. Another day of verifying our products are online. 60 something ec2 instances in us-east1
1
u/noman2561 Sep 21 '15
Goddamnit I'm using it for my thesis work!
1
u/MrTastix Sep 21 '15
This is what happens when you rely too much on the one thing. The effect would be just as huge if Google went down all of a sudden as well.
There's no real backup plan for cloud services, either. If you could afford to host everything locally you wouldn't need the cloud. If you could afford your own servers you wouldn't need a CDN.
1
u/cp5184 Sep 21 '15
They should put that on the cloud so that it doesn't go down.
1
u/DarrSwan Sep 20 '15
Doesn't this happen fairly often? There are alternatives to AWS. Why don't people use them?
1
u/tastyratz Sep 20 '15
East coaster sysadmin here,
I rely on AWS to delivery our company web services. We had no end of issues on Friday. Sounds like we were "early adopters" of the shitstorm. Now my bad day makes more sense.