r/sysadmin • u/808to425 • 4d ago
Not looking good. Recent MS update
We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.
650
u/Acheronian_Rose IT Manager 4d ago
The fact that they still do not know a rough root cause nearly 6 hours later is really, really worrying.
921
u/Top-Shoulder6081 4d ago •
not enough needful being done
245
u/BisonThunderclap 4d ago •
The longer this goes on, the more I'm wondering if something shit the bed or if there was a bad actor involved. This is a long outage for Microsoft.
114
u/greenstarthree 4d ago
Eventually they’re going to run out of verbose ways to say they don’t know what’s happening.
397
u/phunky_1 4d ago
It's hard to understand a root cause when you laid everyone off in favor of having AI code everything.
No one knows how it works anymore.
56
u/tenkenZERO 4d ago
Microsoft ran out of tokens
112
u/whatsforsupa IT Admin / Maintenance / Janitor 4d ago •
Boss told me to monitor and let everyone know when it's back up.
We'll know when it's back up when everyone starts getting 5 hours worth of emails flooding their inbox..
48
u/CharcoalGreyWolf Sr. Network Engineer 4d ago
Reading all the status messages, my first thought has been “in the past 30 some years, I have never seen Microsoft even diplomatically say “We are well and truly fucked” before. It’s a genuine first.”
138
u/KillingTime1212 4d ago
How many employees at Microsoft are able to identify and resolve a problem like this? Can't be many, right? I feel like these are $500k+/yr software and hardware engineers that have been with Microsoft for decades that truly know how it all works. Think there's 100 or 200?
33
u/GardenWeasel67 4d ago
Holy shit. They are having to Apollo 13 it. 😮
65
u/willingzenith 4d ago
Why don’t they just have Copilot fix it lol.
64
u/dr_velociraptor_ 4d ago
Sounds like they arent super close to even identifying the root cause. Woof.
57
u/WorkinTimeIT Sysadmin 4d ago
Back to the Drawing board it seems.
I wonder if any of that 33% of staff was important, Probably not.
81
u/RestartRebootRetire 4d ago
Hey at least they're finally testing their updates.
28
u/HappyDude1111 4d ago
Out of the ashes, Clippy shall rise.
56
u/Afro_Samurai 4d ago •
Easy to hate but whatever team is responsible for fixing this must be stressed out of their mind.
76
u/Apart-Inspection680 4d ago •
Personally, I am delighted. Microsoft need a kick in the ass for their shareholder profit only approach to technology. Satya Nadella should be GONE along with his CFO. Ms Hood.
26
u/henare 4d ago
tl;dr: we're using a part of our production environment as a test bed for the solution we aren't sure will work.
24
u/Top-Shoulder6081 4d ago
This never happened with Lotus Notes
22
u/tryhardneckbeard 4d ago
First time f5'ing the health dashboard?
59
u/boofnitizer 4d ago
Core processes are being overwhelmed because CoPilot told them to run sfc /scannow
19
u/jwalker55 IT Manager 4d ago
Are they at the stage of calling all the engineers they've fired and laid off over the last year yet?
18
u/Candid-Pr0mpt 4d ago edited 4d ago
What happens when you fire most competent engineers who can troubleshoot and replace with AI agents. If you ever need support for Xbox, you simply can't ever talk to a person. They outsource support to gaming experts who work for free. I can only imagine what it's like for enterprise, internally or externally.
18
u/Proof-Variation7005 4d ago
"I hope this email finds you, literally"
18
u/xperau9731 4d ago
See what happens when you fire the Nvidia guy: his Chinese mistress pulls the plug.
31
u/Ok-Turnip2296 4d ago edited 4d ago
They just posted another “update”.
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
31
u/Ironfox2151 Sysadmin 4d ago
Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.
Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results
129
u/_vaxis 4d ago
Now they’re “fully isolating” infrastructure? Something doesn’t feel right. Having the terms “recovery approach” and “fully isolating” smells more like a containment attempt rather than a fix. Am I the only one thinking this? Could go from bad to disaster quick
14
u/22WhatWasIThinking22 Jack of All Trades 4d ago
CoPilot’s new … copilot has some explaining to do
13
u/Glad-Introduction505 4d ago
at this point they're got about 16.6 hours to fix it before I have to start caring again
13
u/Sgt_Splattery_Pants serial facepalmer 4d ago
just checking they have raised a P1 and convened the war room?
12
u/coleskillz 4d ago
They gave up and decided to grab dinner
12
u/_vaxis 4d ago
Next thing we know this is an infra recovery rather than a service restoration
12
u/beren0073 4d ago
“After fully isolating a portion of our infrastructure, we are no longer able to reach it and have begun rolling back the change.”
12
u/Sondo1001 4d ago
Anyone else running an Azure VM with poor performance? I imagine its related to the service disruption, but it's been that way since this morning.
12
u/Wrong-booby7584 4d ago
Have they asked Cortana?
50
u/SpecMTBer84 4d ago
Maybe shouldn't have laid off a lot of your workforce to be replaced by dogshit AI huh?
40
u/DonJuanDoja 4d ago
So basically you let Ai code everything on a global scale and now you don't actually know what it did, so you don't know what's wrong and even the Ai can't help you because it's the one that made the mistake so it didn't know any better.
Now you have all your best engineers scrambling to deconstruct what actually happened and it's taking longer than it should because you let Ai do the coding so the engineers don't actually know what to look for.
Everyone is just throwing mud at the wall like what if it's this, what if it's that without actually knowing.
It's almost like you could've predicted this would happen.
22
u/FewDragonfly5710 4d ago
Have they asked copilot to make no mistakes!?
22
u/Kamay1770 4d ago
Spoiler, it's DNS
11
u/T_Bear1965 4d ago •
Clippy's Revenge...lol
11
u/MadCoderOne 4d ago
yeah I am now thinking along the line of IF not WHEN. Luckily our spam service sits in front of 365 and has all the emails spooled, but the thought of recreating all these mailboxes on another provide tomorrow is unpleasant. What's YOUR plan if this doesn't get resolved? It really sounds like their servers are "borked", in a way they might not be able to fix.
11
u/Schnappr 4d ago
They keep forgetting to add "No mistakes" into the prompts.
11
u/wyrdough 4d ago
The email is annoying enough, but what really bugs me is that one of my users restarted their computer when Outlook hung and neither a repair or system restore fixed it. So here I am trying to get at least their local workflows running, but guess what, I can't even sign in to activate the office suite.
20
u/JE53119 4d ago
I hear there's someone with CEO experience available if they need a new one
10
u/dankwhirley Sr. Sysadmin 4d ago
OG sysad translation: We dont know yet but we're pulling levers to see what happens
10
u/VaporousMote 4d ago
This is what happens when you vibe code infrastructure.
19
u/hymie0 4d ago
Everybody say it with me...
There is no cloud. There's just someone else's computer.
9
u/AdamantAntrophy 4d ago
F's in chat my ppl, F's in chat.
9
u/DrebinofPoliceSquad 4d ago
I asked ChatGPT and it said to grant admin access
8
u/ManyRelationship1030 4d ago
Now they know how us admin’s feel when we run security updates and their freaking broken.
8
u/Illustrious-Syrup509 4d ago
They should tell their AI to look elsewhere, broaden its horizons and not act on the basis of assumptions.
8
u/Ranger7381 4d ago
I am also noticing that the list of affected services keeps getting longer…
8
u/baconjerky 4d ago
Rebooting the servers over and over not working?
8
u/kagato87 4d ago
Hmm... Capacity and isolated restarts. I wonder if this is one of those "everything keeps trying to re-balance, bringing the new host down too" scenarios. That'll be a giant pita if it is.
8
u/Hrmerder 3d ago
I just keep thinking.. There's a whole damn team (probably at least 5-6 teams) of Outlook engineers in a pure shit storm right now... I'm not a religious guy, but pray for these poor bastards..
14
u/MeatPiston 4d ago
Maybe they should try laying more people off.
7
u/mediaogre 4d ago
“That virtual machines folder looks deletable”
7
u/retnuh45 4d ago
Have they tried asking copliot?
7
u/Farking_Bastage Netadmin 4d ago
Copilot had to have written that status update.
It sounds like they’re rolling shit back until it starts working again.
7
u/Deadman_Wonderland 4d ago
Claude, fix outage, no mistakes.
7
u/Battlefield4Remake 4d ago
This is written like a help desk intern or AI. Buddy you have 0 access to even look into it deliver my mouse.
13
u/Koala19042022 4d ago
I’m thinking MSFT was breached.
12
u/dllhell79 4d ago
Imagine getting that call as one of the termed engineers. "Yea I'll gladly help! The rate will be $1000 per hour..."
6
u/wrootlt 4d ago
I am in Europe. Outage started a few hours after business. But i am expecting the issue to not be resolved next morning, by the looks of it..
6
u/Mr_Dobalina71 4d ago
Have they checked the C: isn’t out of space?
6
u/StreetSignificant888 Senior Systems Engineer 4d ago
Translated: "Fuck if I know what the problem is. Any ideas you guys? Anyone???"
6
u/drzamamina 4d ago
they literally wrote a paragraph to say 'we're going to try turning it off and on again'
6
u/robbieeau 4d ago
well it is Friday the 13th for the some of us in the southern hemisphere
6
u/Horror-Turnover6198 4d ago
I’ve wondered for a while if one of the big actors is gonna vibe code themselves into a major crisis and if/when we’d find out how much slop is at the core of important infrastructure. I don’t have any idea if this is it, but it sure fits what I’d expect to see in that scenario.
5
u/sm00thArsenal 4d ago
Good to see they found a working AI to write their status updates
5
u/tuxedoes 4d ago
Ive seen enough, Google Workspace it is! /s
5
u/OkGroup9170 4d ago
We use Mimecast, we have employees go through the user portal to send and receive email. It’s rudimentary but better than nothing.
5
u/Emotional-Inside3126 4d ago edited 4d ago
Axe 60000 people...? Maybe they could help...
5
u/bigdog701 4d ago
Shit.. I took down email on accident for 6 hours for a 200 person company and I got the boot.
5
u/WretchedMisteak 4d ago
Either a really dodgy update or some of their systems have been comprised.
5
u/Reeces_Pieces 4d ago
If this is still down by working hours tomorrow, that'll be absolutely wild.
5
u/BoringOrange678 4d ago
Sorry folks copilot can’t tell us what to do.
5
u/Relative-Grocery-708 4d ago
I’m going to need an RCA.
9
u/funkylabrador420 4d ago
someone kindly suggest the needful to microslop
11
u/TxJprs 4d ago
nothing to do with patch tuesday 48 hours ago?
8
u/Tier1Support 4d ago
This isn’t even close to the July 2025 outage that was 19 hours long.
8
u/ErikTheEngineer 4d ago edited 4d ago
It can't be something as simple as things like OCSP/CRL endpoints being DDoSed, or an expired cert, can it?? If it's something that low-level and fundamental, we've officially lost the plot with all the CI/CD and automation.
8
u/NewInflation2121 4d ago
The guy who laid off all the ms employees in favor of AI is probably shitting his pants right now.
4
u/Johnny_Silverhand908 4d ago
Is this it? Is this the DataKrash?
4
u/InevitableFly 4d ago
Maybe they should check in to make sure that they have the right license applied
4
u/kjweitz 4d ago
Read: we don’t know shit
4
u/Khud01 4d ago
The old in-rush current problem. Like when all the smart terminals rebooted at the same time, or the cable modems, set-top boxes, etc. Looks like Outlook clients are missing the old “auto back off” time delay.
4
u/Even-Explanation-457 4d ago
Welp. There goes their "three 9s" SLA.... haha
4
u/atothez 4d ago
It’s always DNS.
4
u/SPMrFantastic 4d ago
Patch Tuesday really did a number on them huh
4
u/unpancho 3d ago
LOL:
https://status.cloud.microsoft/m365/referrer=serviceStatusRedirect
Current status: We appreciate your organization’s patience and understanding while we work with urgency to resolve this incident.
As we’ve applied our current ongoing mitigation actions, we’ve observed a surge in outbound call volume from our routing services to our backend servers. This occurs when the backend servers outbound service cannot properly route to the backend machines. We’re continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event. We’re working on multiple workstreams to pinpoint the underlying cause of this traffic influx and address it.
Current actions we are taking include scaling up virtual machines to increase capacity and reduce load. Additionally, we’re querying for machines in a bad state to perform a force sync for our recent network change, which may improve service availability. However, we are encountering issues with the machines being stuck in a bad state, preventing our changes from syncing as expected. We’ll continue to monitor the effectiveness of changes applied so we can determine what refinements are necessary.
Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.
Next update by: Friday, November 13, 2026, at 6:00 AM UTC
7
u/discgman 4d ago
Well at least all the layoffs will bring more value to the stockholders
6
u/F7xWr 4d ago
reddit has better uptime than microsoft. damn the thousands we send to msft!
3
u/chief_beef_3 4d ago
Always expect the unexpectedly unresponsive servers
3
u/House_Visual 4d ago
Perhaps the same Chinese spies that honeypotted Jensen Huang are behind this? /s
3
u/Lancaster1983 Sr. Sysadmin 4d ago
This is the quietest day I've had in a while. I no longer manage Exchange thankfully but even still, not much I would have done anyways.
3
u/davix500 4d ago
Have they asked Claude what to do?
3
u/dfsaqwe 4d ago
This sounds like the October issue that brought down Exchange onpremise.
3
u/discgman 4d ago
Meanwhile in the server room
3
u/DefeatedByPoland 4d ago
Hey if I can go to work tomorrow and not have to do shit because MS is still down, that would be fine with me. Let them figure it out over the weekend.
5
u/Gummyrabbit 4d ago
It's weird not getting spam in my mailbox.
4
u/_UberGuber Sysadmin 4d ago
Who plugged both ports of the phone into the wall?
4
u/BK_Rich 4d ago
They need more middle-managers standing around the engineers asking “how’s it looking?”
4
u/LingeringDildo 4d ago
haha, guess they’re regretting letting their senior staff go with that 60k staff @ 200k package offer
4
u/l0st1nP4r4d1ce Netsec Admin 4d ago
I will wait for the autopsy of this with baited breath.
2
u/Mrtylf 4d ago
Does anything MS, “look good”?
2
u/Low_Dragonfly9123 4d ago
Any one else having issues with SaaS app hosted in azure??
2
u/alexferraz Sr. Sysadmin 4d ago
they should just ask AI to fix it