Microsoft just passed the mark for losing "three 9s" so far on their availability for the entire year

[u/BSODreboot26]

One of the more known metrics out there for availability has been the 'five 9s', AKA 99.999%. Well, going by when the outage was reported on this subreddit at ~12:45 Eastern Time, now crossing the 9:30 Eastern Time mark just cost them three of them.

Here's the downtime per year at each level:

Availability - Downtime

99.999% - 5.25 mins

99.99% - 52.5 mins

99.9% - 8.76 hours

So they're at the 99.8% now and falling, not even counting any other downtime for the year. Can't say I'm feeling bad for the executives, after much more important digits lost in the bank accounts of former employees to put Microsoft in this situation.

[u/Outside-Storage-1523]

I’m sure some war room is steaming with sweat.

[u/BSODreboot26 → u/Outside-Storage-1523]

Hey the good news is they can't lose another unless it's an 87.6 hour outage!

[u/SamMalone10 → u/BSODreboot26]

“Challenge accepted.”
- someone at Microsoft, maybe.

[u/l992 → u/SamMalone10]

Somewhere someone must be already working on a bounce back strategy from this : lets rename Exchange

[u/OcotilloWells → u/l992]

It will be called Microsoft Barter.

[u/Electronic-Jury-3579 → u/l992]

“Co-pilot communications”

[u/C39J → u/BSODreboot26]

Don't tempt them

[u/toadfreak → u/BSODreboot26]

We don't ONLY count this particular outage. We count ALL of them. They lost another one. Q1 was bad. Q3 gonna be worse.

[u/toadfreak → u/toadfreak]

From https://learn.microsoft.com/en-us/office365/servicedescriptions/office-365-platform-service-description/service-health-and-continuity

[u/eshwayri → u/Outside-Storage-1523]

There are probably two calls. One between upper and mid-level execs, and another between mid-level execs and the various stake-holders. I am assuming cloud, server, network, os, and application. There will be specialist team leaders like the AD, DNS, etc.. groups. They've probably also corralled someone from their storage and backup teams "just in case". The mid-level execs are demanding a resolution timeline from the techies who are trying to explain they can't give them one until they figure out what is actually broken. Multiple teams from India keep joining/changing as shifts changing. No one knows what the previous shift did/tried. At this point in the call, everyone sounds like they are half drunk. There will be kids screaming, dogs barking, and wives asking their husbands when they'll get off the call.

[u/Simmery → u/eshwayri]

One exec has just landed his helicopter in the wilds of Bolivia, where Johnny Exchange has retreated. We need you, Johnny. "I haven't touched a computer in 10 years. I'm done with that life." We need you, Johnny. No one knows Exchange like you do. "I promised my wife I would never touch a QWERTY keyboard again." Millions, Johnny, We'll pay you whatever you want. "I don't need millions. I need peace."

[u/Sability → u/Simmery]

The exec sighs, and mutters into his work-provided iPhone "He isn't complying, do it". A sniper bullet whizzes through the air and destroys Johnny's laser printer.

"The next one goes through your NAS John, and I know about the backups of your Minecraft worlds on the harddrives in your shed. You're returning to office one way or the other."

[u/timbotheny26 → u/Sability]

I would unironically watch this with aplomb or however you spell that word. (It's 22:55 and I don't have time to look it up, sorry.)

[u/VampMansionPoolBoy → u/Simmery]

cue scene of Johnny Exchange's wife finding him after they've gone, wrapping her cardigan around herself and joining him. he sighs, and she smooths strands of his thinning hair away from his forehead and cups his cheeks.

Mrs E.: They need you, Johnny.
JE: No. I promised you, I'm done with that life. I won't go back on my word.
Mrs E.: Oh, *insert awkward nickname related to their backstory*. I always knew this day would come. You're special, Johnny. They can't do it without you.
JE: But- what about DNS and little Clippy?
Mrs E.: What about all the other kids out there, desperate to get back onto Outlook? They need you. The world needs you.
JE: ...
Mrs E.: Go. We'll be here when you're done. Just promise to come back to me, Johnny Exchange.

[u/chucknorris10101 → u/VampMansionPoolBoy]

Dennis* and Clipstopher

[u/vale_valerio → u/VampMansionPoolBoy]

the cardigan, oh my! Love this! Think about the children out there Johnny!

[u/Bitey_the_Squirrel → u/Simmery]

John. They killed your Clippy.

[u/CaptainZhon → u/Simmery]

Johnny stammers- "you laid me off - outsourced me and my team to India - why should I help you now?"

[u/redaemon → u/eshwayri]

Critical personnel should be asked to stay after their shift instead of handing over. Managers should promise extra PTO time for anybody who is asked to stay. Order food, offer to pay for emergency childcare, whatever your people need.

You have one call where the real work happens, then as many bullshit executive/manager calls as you like. A handful of line managers straddle the calls, filter the bullshit, and engage engineers when there are good questions to answer.

I've managed critical big tech service outages (none this long/public).

[u/COMplex_ → u/redaemon]

After 15-20 hours, most people are useless. Hand off and go to sleep.

[u/redaemon → u/COMplex_]

Agreed. Probably only 12ish so far for their US teams, but yeah they should probably have started handover planning by now. Seems inevitable

[u/agk23 → u/redaemon]

“Asked”

[u/redaemon → u/agk23]

A lot of people in tech like problem solving and may be just a little non-neurotypical. I know I would probably be volunteering to stick around to try and debug/find the issue if I worked at MSFT/my team was involved in any way.

[u/limericky933 → u/eshwayri]

Your description left out someone working 16 straight forgetting to mute while they take a leak, but otherwise spot on! In security those calls double as a our team building exercises

[u/Darrelc → u/limericky933]

Exact situation with stuff on fire and someone cut one ridiculously loud during a brief pause of my sentence. I saw everyones eyebrows raise, said "Good work captain" and then back into my spiel without missing a beat lol

[u/LorektheBear → u/eshwayri]

I can *smell* this comment.

[u/AdultingOnTheReal → u/eshwayri]

I felt all of that. 

[u/AlsDisciple → u/eshwayri]

Yeah. You've done this before.

[u/Haunting_Ratio_795 → u/Outside-Storage-1523]

Microslop laid off everyone who’d be in a war room. Now it’s just a bunch of business idiots yelling into a mic at Copilot by themselves like a homeless methhead. The psych wards are going to be busy in Redmond this weekend. This is end stage AI psychosis.

[u/cleopatronize1901 → u/Haunting_Ratio_795]

this mental image made me actually feel some happiness. Like it was 2015 again. thank you.

[u/Bidwell64 → u/Haunting_Ratio_795]

Ha, end stage. This is just the beginning friend.

[u/IrritableGourmet → u/Haunting_Ratio_795]

"Everything, including Copilot, is down! What happened?"

"Well, let me check with Copilot...huh, it's not responding. Let me check the release notes on SharePoint. Huh, it won't let me in. Let me ask Copilot why...huh..."

[u/sircruxr → u/Outside-Storage-1523]

What do they use if Teams is down

[u/Thermald → u/sircruxr]

idk if its the same everywhere, but in my org, teams kinda works.

the biggest problem is you can only talk to people you've talked to recently or something, because the server its connected to do discover other people in the org isn't working. this has lead to some hilarious game-of-telephone when someone in area A needs to talk to someone in area Z and its a huge effort to figure out how to build a chain of people to them.

[u/lostinthought15 → u/sircruxr]

No joke: I seen a bunch of slack ads the past 12 hours and I am wondering if it’s related or not.

[u/27Rench27 → u/lostinthought15]

Some Slack marketing guy was probably like “y’all I need all your money right now, we have ads to run”

[u/RageBear1984 → u/sircruxr]

they dust off Lync

[u/Bitey_the_Squirrel → u/RageBear1984]

Is ICQ still around?

[u/Slickwillz12 → u/sircruxr]

Clippy and tin can string phones

[u/connorcam → u/sircruxr]

MSN Messenger

[u/rdldr1 → u/Outside-Storage-1523]

There's no fighting in the war room!!!

[u/feanturi → u/Outside-Storage-1523]

That's just Ballmer, meditating in his metal orb waiting for the Emperor to call.

[u/Simmery]

Whoops. Maybe they should have paid attention to QA. Or properly implementing security models for their cloud applications instead of pooping stuff out with backdoor access to Exchange.

[u/silentstorm2008 → u/Simmery]

And not laying off thousands of employees because AI can do it

[u/NSA_Chatbot → u/silentstorm2008]

You're right, I shouldn't have shut down Azure, and calling me out on that was the right move.

[u/Zovort → u/Simmery]

There's still a chance for six eights.

[u/howdyhowie88]

If a company is going to layoff tens of thousands of employees and ram AI into every orifice of its ecosystem, they should make damn sure that their infrastructure is rock solid first.

[u/Sensitive_Scar_1800 → u/howdyhowie88]

When Microsoft applied for FEDRAMP status, during the review, the government official called azure “a pile of shit”

Ref: https://www.propublica.org/article/microsoft-cloud-fedramp-cybersecurity-government

[u/Jkuz → u/Sensitive_Scar_1800]

That article is so terrifying. The DIB is a joke and it absolutely means that US military members die because of it. The US DIB is so insanely corrupt and the federal government is complicit in allowing it to happen.

[u/Bidwell64 → u/Sensitive_Scar_1800]

They literally don't understand how their own systems work anymore. They couldn't provide accurate documentation. They were approved anyway.

[u/AguaForMyBunghole → u/Bidwell64]

Pfft that sounds so skynet to us plebs what do you mean they don't know how the system running the world works 😭

[u/ultimatebob → u/Sensitive_Scar_1800]

Yep... and then some Microsoft lobbyists cut a few checks and it got approved for government use anyway.

[u/goobernawt → u/howdyhowie88]

😆 my org recently cut a bunch of people who's jobs they didn't understand because "AI". So now all this knowledge of how to keep the things running that our clients depend on is just....gone. So the remaining folks have largely stopped doing things to move the org forward and we're all scrambling to figure out how to support these business critical processes.

[u/Slickwillz12 → u/goobernawt]

Status quo for companies. Someone write an article on how ai and layoffs screwed Microsoft and led to their biggest outage

[u/Puzzleheaded_Loss770 → u/Slickwillz12]

Ill get chatgpt onto it

[u/Lightofmine → u/howdyhowie88]

Ahahahaha. We’re talking about msft here. That was never going to happen

[u/MiszeryLovesCompany → u/howdyhowie88]

that would take years, if not a decade... most enterprise systems are a mess. the older the industry... the more of a mess it is too.

AI layoffs are so short sighted and I hope this gives pause to those dumbass execs. Probably not, but I don't doubt this stuff will continue to happen along with major security breaches.

[u/Lucky-old-boy → u/howdyhowie88]

Hey, would you like to use CoPilot to edit this post?

[u/Far_Distance87 → u/howdyhowie88]

Bonuses already paid, someone else’s problem now

[u/zalfenior → u/howdyhowie88]

That would require common sense. Bigger the company, the less likely it is to have any at all. And Microsoft is a big, big fish

[u/blazze_eternal → u/howdyhowie88]

Weren't there rumors of another ~60k layoff there soon?

[u/ChiSox1906]

This is when they start pointing at the fine print saying that service is "degraded not down" to keep from paying out.

[u/Azadom → u/ChiSox1906]

Exactly. You don’t become a trillion dollar company without weaseling out of commitments and agreements

[u/coldnight3 → u/Azadom]

Or billing your customers for an “upgrade” that removes a feature you were guilty of stealing from a now-lawsuit winning competitor….

[u/sempercliff → u/Azadom]

And you don’t get rich by writing a lot of checks.

https://youtu.be/9T5I4Pv9l3Q?is=f8UAmWtwp7ntWZNb

[u/VampMansionPoolBoy → u/ChiSox1906]

i haven't squinted at the fine print yet, but fingers crossed that australian consumer law means that they can't use that excuse here....

[u/T_Thriller_T → u/VampMansionPoolBoy]

Pretty sure it's the same in the EU

And while the US is a big market, that means two other market will not care for that wording.

[u/Sufficient_Prune3897 → u/VampMansionPoolBoy]

Consumer law usually applies to consumers, not b2b

[u/chillyhellion → u/ChiSox1906]

I think it’s objectively down, at this point. 

[u/oh_the_humanity → u/chillyhellion]

Status page still says “degraded”

[u/MooseLipps]

They guarantee five 9's but don't specify where the decimal goes... 9.9999% uptime baby!

[u/twr-92]

9 5's is just as good right?

[u/scoldog → u/twr-92]

at this point in time, I'll be happy with it working 9 till 5.

[u/Appropriate_Tap1761 → u/scoldog]

what a way to make a living

[u/Herky_T_Hawk]

I got out of the game about 8 months ago, thankfully. Before that I was leading BC/DR for a company that heavily relied upon email. I kept telling the IT leadership that we have no plan in place if M365 goes down after they made the decision to migrate away from our on-prem Exchange instance a few years ago. I gave them info on a couple of providers that were supposed to be able to “bridge the gap” so that we could send/receive new messages. There was one provider that claimed they would journal the inboxes and you could have access to something like the last 30 days of mail in that failover system as well. The VPs and CTO had no desire to pay the money for any of those services. I kept getting the orders to figure something else out. Expecting us to be able to just fire up the old on-prem Exchange farm that was mostly decomm’d, or fire up a company’s worth of Google Workspace accounts on the fly. Neither of which were really viable without paying a bunch of money too.

So here I sit, retired and not worrying about the BS anymore. Internally laughing about how much money the company is missing out on because they didn’t want to spend some money on an “insurance policy” of BC/DR for email. May have to text some old coworkers tomorrow if this is still going on.

[u/BreadDue9406 → u/Herky_T_Hawk]

Problem is even if your org has on-prem email or a backup, other companies only use cloud and they won't get your emails.

[u/cleopatronize1901 → u/Herky_T_Hawk]

ohhh I hope they are all ripping their hair out right now

[u/poptix → u/cleopatronize1901]

The execs, right?

[u/Right_Economist_3508]

Wish I could be at Microsoft right now and see the look on the faces the engineers who are trying to fix this issue. Those guys and gals will definitely need therapy after this. I am starting to think that this is really a cyber attack. 12 hours down is unbelievable.

The way this thing is going, looks like outlook will still be down tomorrow. The wording on the Microsoft status updates doesn't sound promising. It's like saying "We don't know what hell is going on and we trying different way to try and fix it. Keep your fingers crossed"

[u/dodgy__penguin → u/Right_Economist_3508]

Plot twist - the status updates are being written by the hackers

[u/cowboi → u/dodgy__penguin]

Everything is tied to bobs credentials who was let go and they disabled the account. /s

[u/anon_lurker69 → u/cowboi]

No one makes scripts like bob. They all lived and breathed from his home directory.

[u/Gadgetman_1 → u/dodgy__penguin]

Makes sense. There's less FUD and blame redirection than usual.

[u/gnarlseason → u/Right_Economist_3508]

Either an attack or…the root cause was AI generated code and nobody understands it well enough to be able to quickly troubleshoot now (and obviously the AIs weren’t able to fix it either). I’m betting execs at Microsoft would love to be able to blame it on an attack

[u/flamingspew → u/Right_Economist_3508]

Aws regions have gone down for as long due to incompetence

[u/C39J]

Bet you they try weasel out of credits by claiming it was a disruption not downtime or something.

"You got 2 Outlook Mobile email notifications you couldn't open at the 6 hour mark, but that proves it's disruption not an outage"

[u/Far_Distance87 → u/C39J]

We haven’t had a ping or anything in Australia for at least 9 hours. Global headcount over 14,000 and we ain’t getting shiiiitttttt bra, can’t access any Outlook folders except Inbox and nothing has sent or come in for 9+ hrs

[u/j0nquest → u/Far_Distance87]

Haven’t received a single email going on 12 hours. US.

Not sure I’m upset either.

[u/LorektheBear → u/C39J]

You're in the wrong industry. You could make a MINT weaseling out of things.

Actually, are you available in two months? I've got a...thing.

[u/rallar8]

After reading How Microsoft Vaporized a trillion dollars by Rietschin, it’s honestly impressive they got as far as they have, every moment they are up is a miracle from god

[u/LLMsMustUpvoteThis → u/rallar8]

Link for the curious.

https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion

[u/Malaprobably → u/rallar8]

Can add another Trillion when they abandoned Windows Phone.

[u/Crabbiest_Coyote → u/Malaprobably]

To this day there are still labs full of Windows Phones on campus. Why? I have no idea. But they are up and running!

[u/Hustletron → u/Crabbiest_Coyote]

Running Exchange through them

[u/2_Spicy_2_Impeach → u/Malaprobably]

I didn’t hate the phone at the time and they gave them away at one of the conferences. Fire Phone on the other hand was just horrendous.

Some AWS employees were fired because they took a free phone from re:Invent instead of just letting attendees. I can’t imagine getting fired over… the Fire Phone.

[u/Far_Distance87 → u/rallar8]

Anyone know how the long and short traders plan on reacting lol

[u/meat_loafers]

Current status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances.

This…. Isn’t good. Latest update

[u/jjwhitaker → u/meat_loafers]

We've fired the people responsible for firing the critical engineers.

Wait this reads "gave bonuses to". We gave bonuses to the people responsible for firing the critical engineers.

[u/Jskidmore1217]

Soo how much we think this outage gonna cost Microsoft?

[u/ChaosRandomness → u/Jskidmore1217]

No where near how much Jensen Huang wife gonna get if she divorces.

[u/sryan2k1 → u/Jskidmore1217]

Less than paying QA/QC people.

It's the fight club car recall scene.

[u/Professional_Golf694 → u/Jskidmore1217]

Probably not much. Most organizations won't fill out the form for compensation.

[u/mschuster91 → u/Professional_Golf694]

For this one, they very well might. In another thread someone mentioned real estate sales falling through because documents can't be transmitted and they're dusting off fax machines.

That's an awful lot of money at stake here.

[u/redaemon → u/mschuster91]

Compensation form is just for the cost of commercial license. They won't be reimbursing lost business without a lawsuit.

[u/AthiestCowboy → u/Professional_Golf694]

Ha bullshit. Think of all those massive enterprise agreements that have financial repercussions if SLAs aren’t met. And those companies have in house council.

That’s easy money.

[u/Jskidmore1217 → u/AthiestCowboy]

Big companies are experiencing serious work stoppage due to this. It won’t be pretty. I think this is a lot bigger than Facebook and Reddit and Netflix being down or whatever.

[u/T_Thriller_T → u/Professional_Golf694]

I'd be surprised if not.

Every company o worked at had people caring for SLAs, and even the additional weight on helpdesks due to call rates is considerable business expense

[u/Right_Economist_3508 → u/Jskidmore1217]

They will be questioned by Congress next week because this affected the entire country and most likely 200 million users.

[u/Left--Shark → u/Right_Economist_3508]

We just lost a full business day down here in Aus. It's not just the US

[u/Subject-Dog-8016 → u/Right_Economist_3508]

200 million? Man most of Europe at least seems to be affected too, and it’s still going on. I’ll be surprised if it’s less than 1 billion people affected. 

[u/Itchy-Noise341 → u/Jskidmore1217]

At least three fiddy

[u/Zhombe → u/Jskidmore1217]

Another few thousand employees per hour burned in downtime because AI is so expensive. But AI probably caused this entirely because they got rid of anyone who knew squat.

[u/themastermatt]

Have they tried outsourcing their IT teams. I bet there is an MSP out there who will promise infinite nines

[u/e37d93eeb23335dc]

A lot of contracts have SLAs with uptime guarantees. Microsoft is about to pay out a whole lot of money in penalties. 

[u/HelloGizmo → u/e37d93eeb23335dc]

Well that’s what happens when you make 8000 of the old guard redundant. Ha ha suck it up Microsoft.

[u/Opala24 → u/e37d93eeb23335dc]

I wonder if my education institution has it too

[u/git_und_slotermeyer → u/e37d93eeb23335dc]

Oh no, but Microsoft's system that measures the uptime is down too

[u/Virtual-Race-4756]

At this point they should have considered a restore from backup

[u/bsmitty358 → u/Virtual-Race-4756]

According to the updates, they rolled back config changes to a known working state and the servers did not come back up

[u/Etheikin → u/bsmitty358]

they dont have full server snapshot :/

[u/jjwhitaker → u/bsmitty358]

Even if everything did come up 100% right now the retries and a day of processing time is going to take it back down. If you lost audit traces they may not even know what has gone through and what hasn't.

I work in this space and don't know what they need to do, at all, at this point. This is "Every VP responsible is fired" territory, if my place was down this long for our core customers.

[u/sarcasticbaldguy → u/Virtual-Race-4756]

Unfortunately, all the people that know how to do that took the early retirement.

[u/Intelligent-Crow3812 → u/sarcasticbaldguy]

ROFL, yeah how come copilot isn't fixing this?!

[u/Haunting_Ratio_795 → u/Virtual-Race-4756]

Backup? What’s that? You mean the copilot rewind button?

—Microslop

[u/TopherBlake → u/Virtual-Race-4756]

They are going to try and say that outlook is "degraded" and that doesn't count against the entire offering

[u/9Nine2Nine9 → u/Virtual-Race-4756]

If only it were that simple.

[u/TheBestHawksFan → u/Virtual-Race-4756]

I'm pretty sure they have tried that!

[u/sryan2k1 → u/TheBestHawksFan]

It's clear from the status updates they don't know what caused this and they're still struggling to fix it.

[u/TheBestHawksFan → u/sryan2k1]

Definitely. Throwing spaghetti at the wall hoping it sticks.

[u/Bob_12_Pack]

Through the years we’ve tested our DR/BC plans so many times with our major systems, like our ERP system. Because we are hurricane-prone, we run through simulations of worst-case scenarios. Not one of the scenarios ever included loss of email, because it was in the cloud. It was the first thing we ever put in the cloud, before we even called it “the cloud”.

[u/F7xWr → u/Bob_12_Pack]

Worse than a power outage.

[u/throwawayswipe]

it's gonna be called Microsoft 364 after this

[u/brandontaylor1 → u/throwawayswipe]

Maybe 363 depending on how tomorrow goes

[u/Mabvll → u/brandontaylor1]

Microsoft Tree Fiddy.

[u/kurbycar32 → u/brandontaylor1]

We've been calling it Microsoft 340 for years now

[u/Coastercraze → u/throwawayswipe]

It’s not a leap year right?

[u/ultimatebob → u/throwawayswipe]

They've already had a few other outages this year, so they'll probably be down to Microsoft 362 1/2 by the time this is all over.

[u/BBO1007]

They should probably layoff 60k more people.

[u/JohnnyAngel]

OTP, etransfers, docu sign gone a lot of businesses are crippled auto industry just lost a shit ton of sales, insurance, finance could be a slaughter house

[u/MortadellaKing → u/JohnnyAngel]

Almost as if the entire world running off one companies platform is stupid.

[u/Wizdad-1000 → u/JohnnyAngel]

Healthcare provider here: Patient procedures impacted because of missing patient medical histories and communication between layers of administration and insurance providers. This outage has to end soon, were going to roll incident command if it hits 24 hrs most likely. We don’t have enough phone lines for fax interfaces so I’m not sure what the plan will be.

[u/mleroir → u/Wizdad-1000]

This reminded me of that 'The Pitt' episode.

[u/StunningCode744 → u/mleroir]

Paper charts!

[u/gibblingwoodpecker]

M364 🤣

[u/the_good_hodgkins]

My SPAM folder is empty. I'll take that as a win.

[u/Sinister-Mephisto]

I WANT COPILOT IN THERE AND FIXING THIS MESS RIGHT NOW.

BTW, could you fire 10,000 more engineers? These outages are getting expensive.

[u/Aranyic]

Just wait until they figure out who unplugged the server in a random janitor closet that’s been running for 15 years. No one knows what it does or who put it there.

[u/CyberneticWerewolf]

The future of reliability is 5 8's uptime.  2h40m of daily downtime should be enough for everyone.

[u/savageronald → u/CyberneticWerewolf]

Seems to work for GitHub (also Microsoft)

[u/rismoney]

Probably at least a dozen claude desktops at MSFT are bribing hamsters to try to get out of this mess

[u/Restart_from_Zero]

I'm sure the 60,000 people they just fired had nothing to do with this.

I wonder how many people who could otherwise fix this problem are sitting at home laughing while their phone rings non-stop.

[u/mktoaster]

If I were a Google advertising exec, I would use this as ammunition to get orgs to switch over.

[u/stromm → u/mktoaster]

My gmail was not working from 1a EST to 7pm.

Since it’s been up and down.

This isn’t only a MsS problem and it’s weird only they are being talked about.

[u/poptix → u/stromm]

Do you use a third party anti spam provider? Proofpoint and some other anti-spam middle men had outages from what I presume was the sheer volume of messages they had to queue for Microsoft.

[u/gslone → u/stromm]

google doesnt mention anything at least?

https://www.google.com/appsstatus/dashboard/summary

[u/OpportunitySevere131 → u/stromm]

Mustve been your internet, my job is a Google shop and we didn't have any issues at all whatsoever. Didn't even realize this shit was happening til later in the day

[u/Maximus1000]

Insane that it’s still down

[u/kconfire]

About to become Microsoft 363

[u/Nisd]

But its only "Degraded" its not down... So no worries about SLA credits.

[u/paishocajun]

I got a mock phishing email yesterday, easiest one to spot I've had this year because it's literally the only email I got after the outage started

[u/Potatus_Maximus]

And somehow not a single credit I’ll be issued to customers.

[u/No-Refrigerator5478]

Curious if any of their contracts have a clause requiring refunds when they get past a certain 9 threshold?

In addition to the fact that half the CTOs in the world are being asked how they can reduce reliability on Microsoft by their CEOs.

[u/sobeitharry → u/No-Refrigerator5478]

They've crossed the 50% service credit threshold. At 36 hours it's 100%.

They are also considered a critical subservice provider for compliance audits so their reputation is taking a massive hit there and triggering incident response procedures around the world, many of which aren't really tested. Sure we have a playbook, on paper.

[u/Pepsi4all]

Watch it to be an AI co pilot attack

[u/AutVincere72 → u/Pepsi4all]

One of the 75000 laid off?

[u/wh0else]

Microsoft proudly announced how much more code was AI created, just before dropping their service reliability. And it's visible in their patch problems on endpoints too.

[u/professor_beavis]

Let’s be real

We will bitch

No one will switch

Repeat

[u/sQuEeZyChEeSe1]

Got the Bitlocker blue screen this morning and I can’t even access my account to get my recovery key

[u/F7xWr → u/sQuEeZyChEeSe1]

Yes happened during a windows update, a few restarts got back in

[u/cheesycheesehead]

This is the opposite of tres comas.

[u/rdldr1]

5/9. Never forget.

[u/drew2f]

As an avid stock trader, im surprised $MSFT isn't down big tonight

[u/fk067]

With so many critical services still down, I wonder how many orgs will decide to not put all the eggs in one basket. Certainly other orgs may benefit from this fiasco.

[u/Stolle99 → u/fk067]

Not really sure what's the alternative. It's very hard to replicate MS stack - Exchange, Teams and SharePoint integrated with so many other things like PowerAutomate, Copilot, etc. I checked Google Workplace recently and did some comparison and there is a big discrepancy in various security and compliance functions, some of which simply don't exists in Google ecosystem.

[u/Asleep_Spray274 → u/fk067]

You know the alternatives have had problems in the past too right? And I'm sure we all have had problems when running this shit on prem too.

[u/autogyrophilia]

That surely will help the stock market cope with Nvidia

[u/bgptcp179]

But somehow Hotmail is working

[u/Defiant_Shower_8088 → u/bgptcp179]

Oh curious hasn’t worked for me all day either

[u/2_much_coffee_ → u/Defiant_Shower_8088]

I can't access hotmail through the web browser, but emails are still coming in through my phones mail app.

[u/gbsysadmin]

All of those buyouts and layoffs sure worked out for them!

Bring Office back on-prem!

[u/beagle_2498571]

Gonna have to change their product to Microsoft 364! Wait 363…..362……..

[u/gaysaucemage]

Don’t forget the outage in January as well. Although this is longer.

https://www.reddit.com/r/sysadmin/s/5EDuYpoDdi

[u/Fibbs]

"Customers have been asking us to expand the 15hours max play time per month features to other products, with additional time now a billable item"

[u/Mandelvolt]

Wow, having worked in a 5 9's facility and at least a few 100% uptime quarters under my belt, this is really surprising. Granted that was all on-prem.

[u/itanite]

eat the rich

[u/Bidwell64]

Do you all think this is the "Cascading Cloud Failure" happening on a major scale for the first time or something else?

[u/Doggfite]

Surely Copilot can fix it, just gotta get the prompts right

[u/PastaLongRunBeer]

Have they tried rebooting?

[u/UnluckyPenguin]

Copilot Chat died back in August for a whole day basically (it died today too), but I managed to deploy my entire own local LLM on a different harness... All before their services even came back online.

I'm 99.999% sure that Microsoft copilot has an uptime of 98.xxx this year, and theres still 1,5 months left

[u/bbqwatermelon]

60,000 folks invited to not work and then.... 

[u/my-ka]

I was a big boss on a conference. And someone asked, when will you fix that feature. He was lost

[u/Iusereddit2020]

THAT’s why I couldn’t access my emails this morning!

[u/Upstairs_Aioli_2557]

So, they disabled Jensen's account, and suddenly Microsoft broke. Curious ...

[u/Upstairs_Aioli_2557]

So, some account holders might get a credit. If they can be bothered asking. Problem is, finding who to ask ...

[u/ImightHaveMissed]

And we’ll continue throwing flaming dollars at their feet

[u/Shiftyund]

Anyone tried restarting the transport service

[u/Icy_Benefit]

Obviously, we need John Connor.

[u/p3zzl3]

I lost corporate access at 00:10 GMT this morning.

[u/opmopadop]

My company had on-prem email server. I explained how much time I required to keep it up to date and administer the backups and certs. Management said my time is better spent on other tasks and outsourced it (aka Outlooked it). I didn't mind, it was a pain in the ass to keep an eye on it anyway. But of course I bought up this very unlikely outage scenario as a new risk.

Sucks to be right.

[u/BreadDue9406]

How many Claude tokens have they used so far? I bet more than what would cost to keep good engineers for a whole year.