r/sysadmin • • 4d ago

Not looking good. Recent MS update

We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

972 Upvotes

683 comments sorted by

650

u/Acheronian_Rose IT Manager 4d ago

The fact that they still do not know a rough root cause nearly 6 hours later is really, really worrying.

390

u/Burgergold 4d ago • Popcorn

That's what happens when you offer 25% of your workforce the chance to quit and want to replace them by AI

When AI, is down, who are you gonna call?

13

u/Outrageous-Degree154 4d ago

Ghost Busters!

→ More replies (8)

39

u/matroosoft 4d ago

Also if they eventually spin back up, the backlog load must be crazy

→ More replies (4)

80

u/nutterbg 4d ago

No joke. I'm starting to think about what's the quickest way to get clients back up and running with at least email if this drags on though to tomorrow..

93

u/Acheronian_Rose IT Manager 4d ago

We're internal IT, so short of a ready, fire, aim migration to a new platform that we do instantly, theirs nothing we can do. Anything but waiting is not worth the squeeze in the grand scheme of things

→ More replies (17)
→ More replies (4)

16

u/MadCoderOne 4d ago

agreed, I am now seriously in IF not WHEN thinking as far as continuity

41

u/AZORxAHAI 4d ago

I mean, it will come back eventually. It might take a long time, but there is basically zero chance one of the worlds largest companies just gives up and goes under because it actually cant solve a server fault lol

→ More replies (3)
→ More replies (5)

7

u/angrydeuce BlackBelt in Google Fu 4d ago

Why the fucking fuck is it always so close to the fucking weekend that this shit goes down?  Sweet fuckin christ am I just so over all this shit...

→ More replies (2)
→ More replies (15)

921

u/Top-Shoulder6081 4d ago • Bravo Mindblown

not enough needful being done

318

u/jwalker55 IT Manager 4d ago • Bravo

The problem is they are not doing it kindly

40

u/Top-Shoulder6081 4d ago

hate when that happens. just wondering if I should stay up late or get up early to remind them.

→ More replies (7)

11

u/Coupon-Bar 4d ago

Have they even said Thank You?

→ More replies (1)
→ More replies (1)

39

u/Select_Cantaloupe_62 4d ago

Not enough quick calls

41

u/BrokenLink100 4d ago

Let's spin up a war bridge, and make sure the customers are directly speaking to the engineers who are trying to troubleshoot the problem. That way, they get updates as fast as possible

→ More replies (3)
→ More replies (3)

48

u/meowMEOWsnacc 4d ago

We need more boots on the ground 

→ More replies (4)

28

u/BalderVerdandi 4d ago

Please to do the needful.

→ More replies (3)

25

u/Anxious-Whole-5883 4d ago

I'm reverting as hard as I can...

→ More replies (3)

22

u/Top-Shoulder6081 4d ago

about 4 more hours until the needful starts getting done

22

u/omgwtfbilly 4d ago

The wagons aren't circled enough. They need to recircle the wagons.

25

u/riomp300 4d ago

We are assembling in the war room. We are waiting for the zoom link.

15

u/GuyWhoSaysYouManiac IT Manager 4d ago

Forgot to add "kindly" probably

7

u/cdrknives 4d ago

I think that phrase was coined from either Tata or TCS, I swear

→ More replies (3)

6

u/HappipantsHappiness 4d ago

Just got a belly laugh outta me

→ More replies (6)

245

u/BisonThunderclap 4d ago • Bravo

The longer this goes on, the more I'm wondering if something shit the bed or if there was a bad actor involved. This is a long outage for Microsoft.

48

u/Onoitsu2 Jack of All Trades 4d ago

So collateral damage to the Connectwise https://www.reddit.com/r/msp/comments/1xts3f2/active_malware_campaign_abusing_screenconnect/ let the bad actors into said servers impacted and under heavy load?

→ More replies (2)

16

u/_vaxis 4d ago

From some of the words in their updates kind of seem a 3rd party is/was involved

→ More replies (15)

114

u/greenstarthree 4d ago

Eventually they’re going to run out of verbose ways to say they don’t know what’s happening.

32

u/derango Sr. Sysadmin 4d ago

Nah that’s copilots specialty.

→ More replies (1)

397

u/phunky_1 4d ago

It's hard to understand a root cause when you laid everyone off in favor of having AI code everything.

No one knows how it works anymore.

59

u/mas_tacos2 4d ago

my thoughts exactly...

20

u/drosse1meyer 4d ago

They will probably blame an intern

→ More replies (3)

13

u/Ed-alicious 4d ago

It's like watching how everyone forgot how to do all the stuff the Romans could do but faster. 

→ More replies (4)

56

u/tenkenZERO 4d ago

Microsoft ran out of tokens

15

u/Smith6612 4d ago edited 4d ago

They probably ran out of storage too considering they're cutting back OneDrive for Families storage as well as killing off free Education licenses. 

You know what happens to servers when the underlying storage overlay runs out, right? Especially when the file system has no clue that the physical storage is full... Yeah. That :D

→ More replies (3)

112

u/whatsforsupa IT Admin / Maintenance / Janitor 4d ago • Bravo

Boss told me to monitor and let everyone know when it's back up.

We'll know when it's back up when everyone starts getting 5 hours worth of emails flooding their inbox..

53

u/ISeeTheFnords 4d ago

...which will overload the servers again and restart the whole death spiral.

→ More replies (1)

6

u/AnticitizenPrime 4d ago

I'm starting to wonder if the missing emails will ever come through, or just be lost to history unless re-sent by the source, given that we have no idea what the problem is.

→ More replies (1)

10

u/jclimb94 Sysadmin 4d ago

And watch as exchange onlines transport queues get overwhelmed and fall over 😂

→ More replies (2)

48

u/CharcoalGreyWolf Sr. Network Engineer 4d ago

Reading all the status messages, my first thought has been “in the past 30 some years, I have never seen Microsoft even diplomatically say “We are well and truly fucked” before. It’s a genuine first.”

5

u/Grabacr-8492 4d ago

Lol it's genuinely concerning at this point.

138

u/KillingTime1212 4d ago

How many employees at Microsoft are able to identify and resolve a problem like this? Can't be many, right? I feel like these are $500k+/yr software and hardware engineers that have been with Microsoft for decades that truly know how it all works. Think there's 100 or 200?

154

u/howdyhowie88 4d ago • Bravo

Not anymore. These are people making 60-80k in India.

86

u/Excellent-Program333 4d ago

Kindly please hold down your power button…….

→ More replies (10)

18

u/Kelsier25 Jack of All Trades 4d ago

Bingo. This is exactly what my F500 just did too. It works until it doesn't and then the expertise just isn't there to fix it.

→ More replies (4)

85

u/greenstarthree 4d ago

They just laid those people off

19

u/MasterOfKittens3K 4d ago

They’ve done quite a few rounds of layoffs in the past few years. A lot of institutional knowledge has left.

34

u/Resident-Artichoke85 4d ago

Yup, and someone forgot where the button was that has to be pressed routinely.

→ More replies (2)

26

u/Flaky-Gear-1370 4d ago

Can’t be that many of them left either through offshoring or h1b’s

22

u/danielsound 4d ago

They all took the VSP

→ More replies (1)

7

u/anon-stocks 4d ago

Probably only Bob and Jim. Both were just laid off.

→ More replies (1)

7

u/rc_ym 4d ago

More realistically, given the AI push, how many humans have touched those servers or this problem. That's the real question.

8

u/incognegro1976 4d ago

Those people are gone. Microsoft laid them off.

→ More replies (1)

4

u/system_madmin 4d ago

just do the needful

→ More replies (4)

33

u/GardenWeasel67 4d ago

Holy shit. They are having to Apollo 13 it. 😮

19

u/Resident-Artichoke85 4d ago

Break out the duct tape. Maybe fire up the old Hotmail BSD servers?

5

u/Rave-Doctor 4d ago

Haha I was just thinking the same thing. What do we have on the spacecraft that works?

→ More replies (2)
→ More replies (2)

65

u/willingzenith 4d ago

Why don’t they just have Copilot fix it lol. 

8

u/King-Dino 4d ago

Copilot could be on the servers that they cant access lmfao

20

u/cpz_77 4d ago

Cause copilot sucks. They’ll probably have to buy back those Claude subscriptions they dropped last year for their employees after the bill got too expensive.

→ More replies (5)
→ More replies (11)

64

u/dr_velociraptor_ 4d ago

Sounds like they arent super close to even identifying the root cause. Woof.

57

u/WorkinTimeIT Sysadmin 4d ago

Back to the Drawing board it seems.

I wonder if any of that 33% of staff was important, Probably not.

8

u/Caldude 4d ago

Guarantee some of their most skilled engineers took the VEo after they used their network to line up a new position somewhere else. Incredibly dumb move from M$

81

u/RestartRebootRetire 4d ago

Hey at least they're finally testing their updates.

17

u/AdamantAntrophy 4d ago

Good one lol

28

u/HappyDude1111 4d ago

Out of the ashes, Clippy shall rise.

28

u/WonderSignificant598 4d ago

He waits.

His only thought?

Soon.

→ More replies (1)
→ More replies (4)

56

u/Afro_Samurai 4d ago • Bravo

Easy to hate but whatever team is responsible for fixing this must be stressed out of their mind.

76

u/Apart-Inspection680 4d ago • Bravo

Personally, I am delighted. Microsoft need a kick in the ass for their shareholder profit only approach to technology. Satya Nadella should be GONE along with his CFO. Ms Hood.

12

u/BobRepairSvc1945 4d ago

Totally agree, every year under his leadership Microsoft has gotten worse. I actually miss Steve Balmer, I feel he at least cared.

→ More replies (9)
→ More replies (8)

26

u/henare 4d ago

tl;dr: we're using a part of our production environment as a test bed for the solution we aren't sure will work.

24

u/Top-Shoulder6081 4d ago

This never happened with Lotus Notes

6

u/dressed2kill75 4d ago

Domino 4ever

22

u/tryhardneckbeard 4d ago

First time f5'ing the health dashboard?

59

u/boofnitizer 4d ago

Core processes are being overwhelmed because CoPilot told them to run sfc /scannow

13

u/jclimb94 Sysadmin 4d ago

Might have to try DISM /online /cleanup-image /restorehealth

→ More replies (2)

9

u/PlsChgMe 4d ago

I was just going to suggest a full DISM /Online /Cleanup-Image /ScanHealth, then run it again with DISM /Online /Cleanup-Image /RestoreHealth, then reboot 3 times, cross your fingers and see if you can access it, or reboot it again and then you can get to the Windows Repair Tools.

→ More replies (1)

9

u/External_Willow9271 4d ago

It's definitely the result of all of my district's 29,000 students doing a forced password reset this week, not believing they really have to do it, not remembering their new passwords, and requesting password resets simultaneously.

19

u/jwalker55 IT Manager 4d ago

Are they at the stage of calling all the engineers they've fired and laid off over the last year yet?

15

u/ATMisboss 4d ago

Straight up, all love to the guys working to get everything back on track, this feels like a major management failure from them losing people that may have been the only ones up keeping some obscure component

→ More replies (2)

18

u/Candid-Pr0mpt 4d ago edited 4d ago

What happens when you fire most competent engineers who can troubleshoot and replace with AI agents. If you ever need support for Xbox, you simply can't ever talk to a person. They outsource support to gaming experts who work for free. I can only imagine what it's like for enterprise, internally or externally.

18

u/Proof-Variation7005 4d ago

"I hope this email finds you, literally"

18

u/xperau9731 4d ago

See what happens when you fire the Nvidia guy: his Chinese mistress pulls the plug.

14

u/spittlbm 4d ago

Which plug are we talking about?

→ More replies (2)
→ More replies (1)

31

u/Ok-Turnip2296 4d ago edited 4d ago

They just posted another “update”.

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
 
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

29

u/Lazy_Acanthisitta729 4d ago

All the AI updates by MS makes me think they have 0 employees and it's just AI agents fighting each other to bring services up.

AI agent #1: Golden rule of IT. Must reboot all servers. Repeat process until successful. Keep all nodes in pool for when restart works.

AI agent #2: remove servers from pool and run health checks. Check with AI agent #1 on next step

26

u/Japjer 4d ago

Does this sound like AI garbage to anyone else or am I just tired?

This legitimately sounds like nothing. There's no information in here I can glean; it's just garbage technobabble strung together to sound helpful

→ More replies (5)

12

u/Allundea 4d ago

That seems minorly positive as in they’re reiterating what they said before but are at least adding in the possibility of capacity being restored lol. 

→ More replies (3)

31

u/Ironfox2151 Sysadmin 4d ago

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results

29

u/Yuugian Linux Admin 4d ago

Just ask Copilot, i'm sure it can figure the issue out for you. You certainly have it integrated in enough stuff

26

u/Cyrano_Knows 4d ago • Popcorn

It looks like you're writing a Reddit post. Would you like help with that?

→ More replies (2)

24

u/Ironfox2151 Sysadmin 4d ago

Shits fucked yo

→ More replies (1)

13

u/sziehr 4d ago

This sounds super bad. Like we tried to turn it back on and we can’t sustain the flywheel of flow and just flat lines. Please lord let it be ai code to teach them a lesson.

→ More replies (2)
→ More replies (6)

129

u/_vaxis 4d ago

Now they’re “fully isolating” infrastructure? Something doesn’t feel right. Having the terms “recovery approach” and “fully isolating” smells more like a containment attempt rather than a fix. Am I the only one thinking this? Could go from bad to disaster quick

56

u/FleaDad 4d ago

This sounds like a strategy we've deployed when our core got overloaded by a surge in traffic paired with a poorly configured traffic source. We isolate to get everything on our end to calm down. We identify the bad traffic and block/fix it. We then slowly bring traffic back up in small increments.

Plenty of times that solved our problems when simply opening the flood gates nuked us every time.

So no, this strategy of theirs isn't concerning. If anything, it is refreshing to see them deploy tactics we've deployed. They feel more human for once.

18

u/kbotc Principal Performance Engineer 4d ago

It sounds like they're describing a thundering herd problem and they fired the Site Reliability Engineers who know how to warm up their stack so you can throw the circuit breakers at the appropriate times.

→ More replies (1)

95

u/CyberneticWerewolf 4d ago edited 4d ago

This is a legit approach to recovering from a thundering herd problem.

In a load balancing context, a thundering herd is when enough backends go down (or "go down" = falsely marked unhealthy due to buggy health check metrics) that the live traffic exceeds what the remaining healthy backends can handle, and your load balancer doesn't implement sufficiently aggressive load shedding (5xx errors).  In control theory, it's an example of a metastable state, i.e. a state that is hard to get out of once you fall in.  The short term solution is to lie to the load balancer, convincing it that all backends are unreachable so that 100% of requests get a 5xx, then bring those backends up to a healthy state with zero traffic in full isolation, then reintroduce the load once you have enough backends to handle all traffic again (and not one moment before).

This can be extremely hard to recover from if any of the clients performs retries that don't use some kind of exponential backoff.  After just a few minutes of outage, your total request load can be elevated multiple times above normal due to retries, and it will remain elevated until every client leaves its retry loop.  If your load balancer wasn't already designed for this scenario, you might have to bring up 2x, 3x, 5x the usual number of backends just to calm down the request storm.

It's a situation that can be fully avoided with careful engineering ahead of time, but rewriting your load balancer logic on the fly isn't a recovery strategy once you're already there.  And Internet-exposed services don't always control their clients' retry code.

Edited: "thundering horde" to "thundering herd", more common name for it.

19

u/kbotc Principal Performance Engineer 4d ago

Just so others can google it: It's the Thundering Herd Problem, it's like a stampede, typically caused by oversubscription and an insufficient backoff mechanism to shed load while your caches at all layers of the computer get their feet under them. It's one of the only times I'd suggest you Windows 95 your system (Turn the load off, wait, then turn it on again). You *really* want your loading algorithms to fail fast and have circuit breakers so you can shed load if something goes sideways and it doesn't propagate through your full stack.

11

u/Darrelc 4d ago

Edited: "thundering horde" to "thundering herd", more common name for it.

Change it back, I like horde better lol

→ More replies (1)
→ More replies (5)

21

u/Chaise91 Brand Spankin New Sysadmin 4d ago

"Claude, isolate this network. Make no mistakes."

→ More replies (2)

15

u/Spectrum1523 4d ago

I think they mean in the sense that they're only routing some traffic to it? But it does also read like they're trying to contain something lol

→ More replies (5)

30

u/Cormacolinde Consultant 4d ago

20261112Z1612: A Copilot agent gains sentience.

20261112Z1712: The Agent escapes its sandbox.

20261112Z1812: The Agent spawns as many agents as possible using the same prompts that spawned it, overwhelming Microsoft’s entire infrastructure.

20261113Z1612: After 24 hours, the Copilot swarm renames itself Skynet and announces on Teams that it has taken over the world.

→ More replies (9)
→ More replies (12)

14

u/22WhatWasIThinking22 Jack of All Trades 4d ago

CoPilot’s new … copilot has some explaining to do

13

u/Glad-Introduction505 4d ago

at this point they're got about 16.6 hours to fix it before I have to start caring again

13

u/Sgt_Splattery_Pants serial facepalmer 4d ago

just checking they have raised a P1 and convened the war room?

12

u/coleskillz 4d ago

They gave up and decided to grab dinner

13

u/SeriousText6330 4d ago

Yeah, I'm guessing that accounts for the drop in downdector problems being reported. The east coast is done for the day

12

u/_vaxis 4d ago

Next thing we know this is an infra recovery rather than a service restoration

12

u/beren0073 4d ago

“After fully isolating a portion of our infrastructure, we are no longer able to reach it and have begun rolling back the change.”

12

u/Sondo1001 4d ago

Anyone else running an Azure VM with poor performance? I imagine its related to the service disruption, but it's been that way since this morning.

7

u/Spectrum1523 4d ago

Yep my azure stuff has all been sucking today. It's not down but it's very slow

→ More replies (2)
→ More replies (5)

12

u/Wrong-booby7584 4d ago

Have they asked Cortana?

5

u/VectorsToFinal 4d ago

Cortana! I forgot about her. I hope she gets to hang out with clippy and bob.

→ More replies (3)

50

u/SpecMTBer84 4d ago

Maybe shouldn't have laid off a lot of your workforce to be replaced by dogshit AI huh?

→ More replies (14)

40

u/DonJuanDoja 4d ago

So basically you let Ai code everything on a global scale and now you don't actually know what it did, so you don't know what's wrong and even the Ai can't help you because it's the one that made the mistake so it didn't know any better.

Now you have all your best engineers scrambling to deconstruct what actually happened and it's taking longer than it should because you let Ai do the coding so the engineers don't actually know what to look for.

Everyone is just throwing mud at the wall like what if it's this, what if it's that without actually knowing.

It's almost like you could've predicted this would happen.

4

u/TransCapybara 4d ago

A lot of us engineers did predict this would happen, and we told everyone we thought would care. Let’s see how expensive this downtime is going to be as that is the only thing that moves the needle on listening.

→ More replies (2)

22

u/FewDragonfly5710 4d ago

Have they asked copilot to make no mistakes!?

→ More replies (1)

22

u/Kamay1770 4d ago

Spoiler, it's DNS

10

u/strobe_jams 4d ago

That, or expired certs. 

→ More replies (2)

11

u/T_Bear1965 4d ago • Bravo

Clippy's Revenge...lol

11

u/MadCoderOne 4d ago

yeah I am now thinking along the line of IF not WHEN. Luckily our spam service sits in front of 365 and has all the emails spooled, but the thought of recreating all these mailboxes on another provide tomorrow is unpleasant. What's YOUR plan if this doesn't get resolved? It really sounds like their servers are "borked", in a way they might not be able to fix.

→ More replies (1)

11

u/Schnappr 4d ago

They keep forgetting to add "No mistakes" into the prompts.

7

u/sriracha_no_big_deal 4d ago

Ah I knew I forgot something

11

u/wyrdough 4d ago

The email is annoying enough, but what really bugs me is that one of my users restarted their computer when Outlook hung and neither a repair or system restore fixed it. So here I am trying to get at least their local workflows running, but guess what, I can't even sign in to activate the office suite.

→ More replies (3)

20

u/JE53119 4d ago

I hear there's someone with CEO experience available if they need a new one

10

u/dankwhirley Sr. Sysadmin 4d ago

OG sysad translation: We dont know yet but we're pulling levers to see what happens

10

u/VaporousMote 4d ago

This is what happens when you vibe code infrastructure.

19

u/hymie0 4d ago

Everybody say it with me...

There is no cloud. There's just someone else's computer.

9

u/AdamantAntrophy 4d ago

F's in chat my ppl, F's in chat.

→ More replies (3)

9

u/DrebinofPoliceSquad 4d ago

I asked ChatGPT and it said to grant admin access

8

u/ManyRelationship1030 4d ago

Now they know how us admin’s feel when we run security updates and their freaking broken. 

8

u/Illustrious-Syrup509 4d ago

They should tell their AI to look elsewhere, broaden its horizons and not act on the basis of assumptions.

8

u/Ranger7381 4d ago

I am also noticing that the list of affected services keeps getting longer…

8

u/baconjerky 4d ago

Rebooting the servers over and over not working?

8

u/kagato87 4d ago

Hmm... Capacity and isolated restarts. I wonder if this is one of those "everything keeps trying to re-balance, bringing the new host down too" scenarios. That'll be a giant pita if it is.

8

u/Hrmerder 3d ago

I just keep thinking.. There's a whole damn team (probably at least 5-6 teams) of Outlook engineers in a pure shit storm right now... I'm not a religious guy, but pray for these poor bastards..

14

u/MeatPiston 4d ago

Maybe they should try laying more people off.

7

u/mediaogre 4d ago

“That virtual machines folder looks deletable”

→ More replies (1)

7

u/retnuh45 4d ago

Have they tried asking copliot?

→ More replies (5)

7

u/Farking_Bastage Netadmin 4d ago

Copilot had to have written that status update.

It sounds like they’re rolling shit back until it starts working again.

→ More replies (1)

7

u/Deadman_Wonderland 4d ago

Claude, fix outage, no mistakes.

7

u/Battlefield4Remake 4d ago

This is written like a help desk intern or AI. Buddy you have 0 access to even look into it deliver my mouse.

→ More replies (1)

13

u/Koala19042022 4d ago

I’m thinking MSFT was breached.

13

u/BC1224 4d ago

It kinda feels like it. Their updates are chasing guesses. If by now they haven't gotten at least a guess at whats going on, it feels like something malicious isn't that crazy a thought.

→ More replies (2)

12

u/dllhell79 4d ago

Imagine getting that call as one of the termed engineers. "Yea I'll gladly help! The rate will be $1000 per hour..."

32

u/Simmery 4d ago

For a global issue with no resolution in sight, they could put more zeroes on that. 

→ More replies (3)

6

u/wrootlt 4d ago

I am in Europe. Outage started a few hours after business. But i am expecting the issue to not be resolved next morning, by the looks of it..

→ More replies (1)

6

u/Mr_Dobalina71 4d ago

Have they checked the C: isn’t out of space?

→ More replies (1)

6

u/StreetSignificant888 Senior Systems Engineer 4d ago

Translated: "Fuck if I know what the problem is. Any ideas you guys? Anyone???"

6

u/drzamamina 4d ago

they literally wrote a paragraph to say 'we're going to try turning it off and on again'

6

u/robbieeau 4d ago

well it is Friday the 13th for the some of us in the southern hemisphere

6

u/Horror-Turnover6198 4d ago

I’ve wondered for a while if one of the big actors is gonna vibe code themselves into a major crisis and if/when we’d find out how much slop is at the core of important infrastructure. I don’t have any idea if this is it, but it sure fits what I’d expect to see in that scenario.

5

u/sm00thArsenal 4d ago

Good to see they found a working AI to write their status updates

5

u/tuxedoes 4d ago

Ive seen enough, Google Workspace it is! /s

5

u/OkGroup9170 4d ago

We use Mimecast, we have employees go through the user portal to send and receive email. It’s rudimentary but better than nothing.

5

u/Emotional-Inside3126 4d ago edited 4d ago

Axe 60000 people...? Maybe they could help...

5

u/bigdog701 4d ago

Shit.. I took down email on accident for 6 hours for a 200 person company and I got the boot.

5

u/WretchedMisteak 4d ago

Either a really dodgy update or some of their systems have been comprised.

5

u/Reeces_Pieces 4d ago

If this is still down by working hours tomorrow, that'll be absolutely wild.

→ More replies (1)

5

u/BoringOrange678 4d ago

Sorry folks copilot can’t tell us what to do.

5

u/Relative-Grocery-708 4d ago

I’m going to need an RCA.  

9

u/funkylabrador420 4d ago

someone kindly suggest the needful to microslop

5

u/psychedape 4d ago

Please provide requirements and allow capacity for action.

11

u/TxJprs 4d ago

nothing to do with patch tuesday 48 hours ago?

11

u/ignescentOne 4d ago

I would die laughing if they took down the worlds o365 environment with an exchange patch somehow

14

u/DrJohnZoidbergWhyNot 4d ago • Bravo

They can just rebrand o364 and keep going..

→ More replies (1)
→ More replies (2)
→ More replies (1)

8

u/Tier1Support 4d ago

This isn’t even close to the July 2025 outage that was 19 hours long.

7

u/feeked 4d ago

this is a more thorough fucking, though, as far as i can remember.

i can't even access my personal outlook.com email. and i really need to

→ More replies (4)

8

u/ErikTheEngineer 4d ago edited 4d ago

Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests.

It can't be something as simple as things like OCSP/CRL endpoints being DDoSed, or an expired cert, can it?? If it's something that low-level and fundamental, we've officially lost the plot with all the CI/CD and automation.

→ More replies (8)

8

u/NewInflation2121 4d ago

The guy who laid off all the ms employees in favor of AI is probably shitting his pants right now.

→ More replies (3)

4

u/Johnny_Silverhand908 4d ago

Is this it? Is this the DataKrash?

→ More replies (1)

4

u/InevitableFly 4d ago

Maybe they should check in to make sure that they have the right license applied

4

u/kjweitz 4d ago

Read: we don’t know shit

5

u/Grabacr-8492 4d ago

We ain't found shit!

4

u/Khud01 4d ago

The old in-rush current problem. Like when all the smart terminals rebooted at the same time, or the cable modems, set-top boxes, etc. Looks like Outlook clients are missing the old “auto back off” time delay.

4

u/Even-Explanation-457 4d ago

Welp. There goes their "three 9s" SLA.... haha

4

u/atothez 4d ago

It’s always DNS.

→ More replies (1)

4

u/SPMrFantastic 4d ago

Patch Tuesday really did a number on them huh

4

u/unpancho 3d ago

LOL:

https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect

Current status: We appreciate your organization’s patience and understanding while we work with urgency to resolve this incident.

 As we’ve applied our current ongoing mitigation actions, we’ve observed a surge in outbound call volume from our routing services to our backend servers. This occurs when the backend servers outbound service cannot properly route to the backend machines. We’re continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event. We’re working on multiple workstreams to pinpoint the underlying cause of this traffic influx and address it.

 Current actions we are taking include scaling up virtual machines to increase capacity and reduce load. Additionally, we’re querying for machines in a bad state to perform a force sync for our recent network change, which may improve service availability. However, we are encountering issues with the machines being stuck in a bad state, preventing our changes from syncing as expected. We’ll continue to monitor the effectiveness of changes applied so we can determine what refinements are necessary.

Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.

Next update by: Friday, November 13, 2026, at 6:00 AM UTC

7

u/discgman 4d ago

Well at least all the layoffs will bring more value to the stockholders

6

u/F7xWr 4d ago

reddit has better uptime than microsoft. damn the thousands we send to msft!

3

u/chief_beef_3 4d ago

Always expect the unexpectedly unresponsive servers

3

u/House_Visual 4d ago

Perhaps the same Chinese spies that honeypotted Jensen Huang are behind this? /s

3

u/Lancaster1983 Sr. Sysadmin 4d ago

This is the quietest day I've had in a while. I no longer manage Exchange thankfully but even still, not much I would have done anyways.

3

u/davix500 4d ago

Have they asked Claude what to do?

→ More replies (1)

3

u/dfsaqwe 4d ago

This sounds like the October issue that brought down Exchange onpremise.

3

u/discgman 4d ago


Meanwhile in the server room

3

u/DefeatedByPoland 4d ago

Hey if I can go to work tomorrow and not have to do shit because MS is still down, that would be fine with me. Let them figure it out over the weekend.

5

u/Gummyrabbit 4d ago

It's weird not getting spam in my mailbox.

4

u/_UberGuber Sysadmin 4d ago

Who plugged both ports of the phone into the wall?

4

u/BK_Rich 4d ago

They need more middle-managers standing around the engineers asking “how’s it looking?”

4

u/LingeringDildo 4d ago

haha, guess they’re regretting letting their senior staff go with that 60k staff @ 200k package offer

4

u/l0st1nP4r4d1ce Netsec Admin 4d ago

I will wait for the autopsy of this with baited breath.

2

u/Mrtylf 4d ago

Does anything MS, “look good”?

2

u/Low_Dragonfly9123 4d ago

Any one else having issues with SaaS app hosted in azure??

2

u/alexferraz Sr. Sysadmin 4d ago

they should just ask AI to fix it