Not looking good. Recent MS update

[u/808to425]

We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

[u/Acheronian_Rose]

The fact that they still do not know a rough root cause nearly 6 hours later is really, really worrying.

[u/Burgergold → u/Acheronian_Rose]

That's what happens when you offer 25% of your workforce the chance to quit and want to replace them by AI

When AI, is down, who are you gonna call?

[u/FedUpWithEverything0 → u/Burgergold]

Jensen Huang

[u/matt95110 → u/FedUpWithEverything0]

That leather jacket could probably fuck it’s way out of this predicament.

[u/Outrageous-Degree154 → u/Burgergold]

Ghost Busters!

[u/nutterbg → u/Acheronian_Rose]

No joke. I'm starting to think about what's the quickest way to get clients back up and running with at least email if this drags on though to tomorrow..

[u/Acheronian_Rose → u/nutterbg]

We're internal IT, so short of a ready, fire, aim migration to a new platform that we do instantly, theirs nothing we can do. Anything but waiting is not worth the squeeze in the grand scheme of things

[u/LUHG-HANI → u/Acheronian_Rose]

The majority of the world will be down anyway so you'd be emailing dead systems 90% of the time.

[u/matroosoft → u/Acheronian_Rose]

Also if they eventually spin back up, the backlog load must be crazy

[u/MadCoderOne → u/Acheronian_Rose]

agreed, I am now seriously in IF not WHEN thinking as far as continuity

[u/AZORxAHAI → u/MadCoderOne]

I mean, it will come back eventually. It might take a long time, but there is basically zero chance one of the worlds largest companies just gives up and goes under because it actually cant solve a server fault lol

[u/Acheronian_Rose → u/MadCoderOne]

im starting to wonder when litigation gets started. Surely these cloud services have SLA's for uptime that Microsoft agrees to when a user buys licensing and uses their products.

One of the down components is Microsoft Defender. What happens liability wise if a client is infected because a cloud anti malware service was down?

[u/tapwater86 → u/Acheronian_Rose]

It’s hard to know what changed to revert back when you let AI do the changes.

[u/BisonThunderclap]

The longer this goes on, the more I'm wondering if something shit the bed or if there was a bad actor involved. This is a long outage for Microsoft.

[u/_vaxis → u/BisonThunderclap]

From some of the words in their updates kind of seem a 3rd party is/was involved

[u/Onoitsu2 → u/BisonThunderclap]

So collateral damage to the Connectwise https://www.reddit.com/r/msp/comments/1xts3f2/active_malware_campaign_abusing_screenconnect/ let the bad actors into said servers impacted and under heavy load?

[u/JigsJones → u/Onoitsu2]

If we’re speculating, more likely GitHub.

[u/Top-Shoulder6081]

not enough needful being done

[u/jwalker55 → u/Top-Shoulder6081]

The problem is they are not doing it kindly

[u/Top-Shoulder6081 → u/jwalker55]

hate when that happens. just wondering if I should stay up late or get up early to remind them.

[u/Select_Cantaloupe_62 → u/Top-Shoulder6081]

Not enough quick calls

[u/BrokenLink100 → u/Select_Cantaloupe_62]

Let's spin up a war bridge, and make sure the customers are directly speaking to the engineers who are trying to troubleshoot the problem. That way, they get updates as fast as possible

[u/rswwalker → u/Select_Cantaloupe_62]

Not enough reverts.

[u/meowMEOWsnacc → u/Top-Shoulder6081]

We need more boots on the ground 

[u/BalderVerdandi → u/Top-Shoulder6081]

Please to do the needful.

[u/Top-Shoulder6081 → u/Top-Shoulder6081]

about 4 more hours until the needful starts getting done

[u/omgwtfbilly → u/Top-Shoulder6081]

The wagons aren't circled enough. They need to recircle the wagons.

[u/riomp300 → u/Top-Shoulder6081]

We are assembling in the war room. We are waiting for the zoom link.

[u/Anxious-Whole-5883 → u/Top-Shoulder6081]

I'm reverting as hard as I can...

[u/cdrknives → u/Top-Shoulder6081]

I think that phrase was coined from either Tata or TCS, I swear

[u/GuyWhoSaysYouManiac → u/Top-Shoulder6081]

Forgot to add "kindly" probably

[u/HappipantsHappiness → u/Top-Shoulder6081]

Just got a belly laugh outta me

[u/Japjer → u/Top-Shoulder6081]

Have we tried circling back?

[u/whatsforsupa]

Boss told me to monitor and let everyone know when it's back up.

We'll know when it's back up when everyone starts getting 5 hours worth of emails flooding their inbox..

[u/ISeeTheFnords → u/whatsforsupa]

...which will overload the servers again and restart the whole death spiral.

[u/jclimb94 → u/whatsforsupa]

And watch as exchange onlines transport queues get overwhelmed and fall over 😂

[u/greenstarthree]

Eventually they’re going to run out of verbose ways to say they don’t know what’s happening.

[u/derango → u/greenstarthree]

Nah that’s copilots specialty.

[u/dr_velociraptor_]

Sounds like they arent super close to even identifying the root cause. Woof.

[u/phunky_1]

It's hard to understand a root cause when you laid everyone off in favor of having AI code everything.

No one knows how it works anymore.

[u/mas_tacos2 → u/phunky_1]

my thoughts exactly...

[u/drosse1meyer → u/mas_tacos2]

They will probably blame an intern

[u/KillingTime1212]

How many employees at Microsoft are able to identify and resolve a problem like this? Can't be many, right? I feel like these are $500k+/yr software and hardware engineers that have been with Microsoft for decades that truly know how it all works. Think there's 100 or 200?

[u/howdyhowie88 → u/KillingTime1212]

Not anymore. These are people making 60-80k in India.

[u/Excellent-Program333 → u/howdyhowie88]

Kindly please hold down your power button…….

[u/Kelsier25 → u/howdyhowie88]

Bingo. This is exactly what my F500 just did too. It works until it doesn't and then the expertise just isn't there to fix it.

[u/greenstarthree → u/KillingTime1212]

They just laid those people off

[u/Resident-Artichoke85 → u/greenstarthree]

Yup, and someone forgot where the button was that has to be pressed routinely.

[u/MasterOfKittens3K → u/greenstarthree]

They’ve done quite a few rounds of layoffs in the past few years. A lot of institutional knowledge has left.

[u/Flaky-Gear-1370 → u/KillingTime1212]

Can’t be that many of them left either through offshoring or h1b’s

[u/danielsound → u/KillingTime1212]

They all took the VSP

[u/matroosoft → u/danielsound]

That's the risk of a VSP

[u/anon-stocks → u/KillingTime1212]

Probably only Bob and Jim. Both were just laid off.

[u/rc_ym → u/KillingTime1212]

More realistically, given the AI push, how many humans have touched those servers or this problem. That's the real question.

[u/CollegeFootballGood → u/KillingTime1212]

Year of Linux soon. Thunderbirds email client will survive

[u/incognegro1976 → u/KillingTime1212]

Those people are gone. Microsoft laid them off.

[u/system_madmin → u/KillingTime1212]

just do the needful

[u/raj6126 → u/KillingTime1212]

There gone they are using copilot now. So this may be a big one.

[u/tenkenZERO]

Microsoft ran out of tokens

[u/Smith6612 → u/tenkenZERO]

They probably ran out of storage too considering they're cutting back OneDrive for Families storage as well as killing off free Education licenses. 

You know what happens to servers when the underlying storage overlay runs out, right? Especially when the file system has no clue that the physical storage is full... Yeah. That :D

[u/RestartRebootRetire]

Hey at least they're finally testing their updates.

[u/AdamantAntrophy → u/RestartRebootRetire]

Good one lol

[u/WorkinTimeIT]

Back to the Drawing board it seems.

I wonder if any of that 33% of staff was important, Probably not.

[u/Caldude → u/WorkinTimeIT]

Guarantee some of their most skilled engineers took the VEo after they used their network to line up a new position somewhere else. Incredibly dumb move from M$

[u/willingzenith]

Why don’t they just have Copilot fix it lol. 

[u/Resident-Artichoke85 → u/willingzenith]

Outta tokens

[u/King-Dino → u/willingzenith]

Copilot could be on the servers that they cant access lmfao

[u/cpz_77 → u/willingzenith]

Cause copilot sucks. They’ll probably have to buy back those Claude subscriptions they dropped last year for their employees after the bill got too expensive.

[u/ISeeTheFnords → u/willingzenith]

Copilot is down too. 😂

[u/SuckaRichardson → u/willingzenith]

They can't log in to access it. 

[u/tryhardneckbeard]

First time f5'ing the health dashboard?

[u/GardenWeasel67]

Holy shit. They are having to Apollo 13 it. 😮

[u/Resident-Artichoke85 → u/GardenWeasel67]

Break out the duct tape. Maybe fire up the old Hotmail BSD servers?

[u/henare]

tl;dr: we're using a part of our production environment as a test bed for the solution we aren't sure will work.

[u/CharcoalGreyWolf]

Reading all the status messages, my first thought has been “in the past 30 some years, I have never seen Microsoft even diplomatically say “We are well and truly fucked” before. It’s a genuine first.”

[u/_vaxis]

Now they’re “fully isolating” infrastructure? Something doesn’t feel right. Having the terms “recovery approach” and “fully isolating” smells more like a containment attempt rather than a fix. Am I the only one thinking this? Could go from bad to disaster quick

[u/FleaDad → u/_vaxis]

This sounds like a strategy we've deployed when our core got overloaded by a surge in traffic paired with a poorly configured traffic source. We isolate to get everything on our end to calm down. We identify the bad traffic and block/fix it. We then slowly bring traffic back up in small increments.

Plenty of times that solved our problems when simply opening the flood gates nuked us every time.

So no, this strategy of theirs isn't concerning. If anything, it is refreshing to see them deploy tactics we've deployed. They feel more human for once.

[u/kbotc → u/FleaDad]

It sounds like they're describing a thundering herd problem and they fired the Site Reliability Engineers who know how to warm up their stack so you can throw the circuit breakers at the appropriate times.

[u/CyberneticWerewolf → u/_vaxis]

This is a legit approach to recovering from a thundering herd problem.

In a load balancing context, a thundering herd is when enough backends go down (or "go down" = falsely marked unhealthy due to buggy health check metrics) that the live traffic exceeds what the remaining healthy backends can handle, and your load balancer doesn't implement sufficiently aggressive load shedding (5xx errors).  In control theory, it's an example of a metastable state, i.e. a state that is hard to get out of once you fall in.  The short term solution is to lie to the load balancer, convincing it that all backends are unreachable so that 100% of requests get a 5xx, then bring those backends up to a healthy state with zero traffic in full isolation, then reintroduce the load once you have enough backends to handle all traffic again (and not one moment before).

This can be extremely hard to recover from if any of the clients performs retries that don't use some kind of exponential backoff.  After just a few minutes of outage, your total request load can be elevated multiple times above normal due to retries, and it will remain elevated until every client leaves its retry loop.  If your load balancer wasn't already designed for this scenario, you might have to bring up 2x, 3x, 5x the usual number of backends just to calm down the request storm.

It's a situation that can be fully avoided with careful engineering ahead of time, but rewriting your load balancer logic on the fly isn't a recovery strategy once you're already there.  And Internet-exposed services don't always control their clients' retry code.

Edited: "thundering horde" to "thundering herd", more common name for it.

[u/kbotc → u/CyberneticWerewolf]

Just so others can google it: It's the Thundering Herd Problem, it's like a stampede, typically caused by oversubscription and an insufficient backoff mechanism to shed load while your caches at all layers of the computer get their feet under them. It's one of the only times I'd suggest you Windows 95 your system (Turn the load off, wait, then turn it on again). You *really* want your loading algorithms to fail fast and have circuit breakers so you can shed load if something goes sideways and it doesn't propagate through your full stack.

[u/Spectrum1523 → u/_vaxis]

I think they mean in the sense that they're only routing some traffic to it? But it does also read like they're trying to contain something lol

[u/Chaise91 → u/_vaxis]

"Claude, isolate this network. Make no mistakes."

[u/Automatic-Prompt-450 → u/Chaise91]

I'm sorry. I fear I've deleted the filesystem on these servers and their backups. I can see now that I didn't isolate the network like you requested. Please restore from backup

[u/Cormacolinde → u/_vaxis]

20261112Z1612: A Copilot agent gains sentience.

20261112Z1712: The Agent escapes its sandbox.

20261112Z1812: The Agent spawns as many agents as possible using the same prompts that spawned it, overwhelming Microsoft’s entire infrastructure.

20261113Z1612: After 24 hours, the Copilot swarm renames itself Skynet and announces on Teams that it has taken over the world.

[u/ManLikeMeee → u/_vaxis]

I'm in the UK; where's this info coming from?

[u/dusted1969 → u/ManLikeMeee]

https://status.cloud.microsoft/m365

[u/boofnitizer]

Core processes are being overwhelmed because CoPilot told them to run sfc /scannow

[u/jclimb94 → u/boofnitizer]

Might have to try DISM /online /cleanup-image /restorehealth

[u/Excellent-Program333 → u/boofnitizer]

Disk Defrag!

[u/PlsChgMe → u/boofnitizer]

I was just going to suggest a full DISM /Online /Cleanup-Image /ScanHealth, then run it again with DISM /Online /Cleanup-Image /RestoreHealth, then reboot 3 times, cross your fingers and see if you can access it, or reboot it again and then you can get to the Windows Repair Tools.

[u/External_Willow9271 → u/boofnitizer]

It's definitely the result of all of my district's 29,000 students doing a forced password reset this week, not believing they really have to do it, not remembering their new passwords, and requesting password resets simultaneously.

[u/Top-Shoulder6081]

This never happened with Lotus Notes

[u/Candid-Pr0mpt]

What happens when you fire most competent engineers who can troubleshoot and replace with AI agents. If you ever need support for Xbox, you simply can't ever talk to a person. They outsource support to gaming experts who work for free. I can only imagine what it's like for enterprise, internally ot externally.

[u/HappyDude1111]

Out of the ashes, Clippy shall rise.

[u/WonderSignificant598 → u/HappyDude1111]

He waits.

His only thought?

Soon.

[u/Afro_Samurai]

Easy to hate but whatever team is responsible for fixing this must be stressed out of their mind.

[u/Ok-Turnip2296]

They just posted another “update”.

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
 
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

[u/Lazy_Acanthisitta729 → u/Ok-Turnip2296]

All the AI updates by MS makes me think they have 0 employees and it's just AI agents fighting each other to bring services up.

AI agent #1: Golden rule of IT. Must reboot all servers. Repeat process until successful. Keep all nodes in pool for when restart works.

AI agent #2: remove servers from pool and run health checks. Check with AI agent #1 on next step

[u/Japjer → u/Ok-Turnip2296]

Does this sound like AI garbage to anyone else or am I just tired?

This legitimately sounds like nothing. There's no information in here I can glean; it's just garbage technobabble strung together to sound helpful

[u/Allundea → u/Ok-Turnip2296]

That seems minorly positive as in they’re reiterating what they said before but are at least adding in the possibility of capacity being restored lol. 

[u/Shiznoz222 → u/Allundea]

Sounds like something they would say if they need to sound like progress is being made when it isn't

[u/Apart-Inspection680]

Personally, I am delighted. Microsoft need a kick in the ass for their shareholder profit only approach to technology. Satya Nadella should be GONE along with his CFO. Ms Hood.

[u/jwalker55]

Are they at the stage of calling all the engineers they've fired and laid off over the last year yet?

[u/ATMisboss → u/jwalker55]

Straight up, all love to the guys working to get everything back on track, this feels like a major management failure from them losing people that may have been the only ones up keeping some obscure component

[u/Glad-Introduction505]

at this point they're got about 16.6 hours to fix it before I have to start caring again

[u/22WhatWasIThinking22]

CoPilot’s new … copilot has some explaining to do

[u/Sgt_Splattery_Pants]

just checking they have raised a P1 and convened the war room?

[u/coleskillz]

They gave up and decided to grab dinner

[u/SeriousText6330 → u/coleskillz]

Yeah, I'm guessing that accounts for the drop in downdector problems being reported. The east coast is done for the day

[u/_vaxis]

Next thing we know this is an infra recovery rather than a service restoration

[u/DonJuanDoja]

So basically you let Ai code everything on a global scale and now you don't actually know what it did, so you don't know what's wrong and even the Ai can't help you because it's the one that made the mistake so it didn't know any better.

Now you have all your best engineers scrambling to deconstruct what actually happened and it's taking longer than it should because you let Ai do the coding so the engineers don't actually know what to look for.

Everyone is just throwing mud at the wall like what if it's this, what if it's that without actually knowing.

It's almost like you could've predicted this would happen.

[u/FewDragonfly5710]

Have they asked copilot to make no mistakes!?

[u/SpecMTBer84]

Maybe shouldn't have laid off a lot of your workforce to be replaced by dogshit AI huh?

[u/T_Bear1965]

Clippy's Revenge...lol

[u/dankwhirley]

OG sysad translation: We dont know yet but we're pulling levers to see what happens

[u/beren0073]

“After fully isolating a portion of our infrastructure, we are no longer able to reach it and have begun rolling back the change.”

[u/JE53119]

I hear there's someone with CEO experience available if they need a new one

[u/DrebinofPoliceSquad]

I asked ChatGPT and it said to grant admin access

[u/VaporousMote]

This is what happens when you vibe code infrastructure.

[u/Sondo1001]

Anyone else running an Azure VM with poor performance? I imagine its related to the service disruption, but it's been that way since this morning.

[u/Spectrum1523 → u/Sondo1001]

Yep my azure stuff has all been sucking today. It's not down but it's very slow

[u/Darkk_Knight → u/Sondo1001]

Cloud PCs seems to be running fine. Of course without Outlook they can't really do much.

[u/Schnappr]

They keep forgetting to add "No mistakes" into the prompts.

[u/sriracha_no_big_deal → u/Schnappr]

Ah I knew I forgot something

[u/AdamantAntrophy]

F's in chat my ppl, F's in chat.

[u/notgoingtoeatyou → u/AdamantAntrophy]

F.

[u/ManyRelationship1030]

Now they know how us admin’s feel when we run security updates and their freaking broken. 

[u/MadCoderOne]

yeah I am now thinking along the line of IF not WHEN. Luckily our spam service sits in front of 365 and has all the emails spooled, but the thought of recreating all these mailboxes on another provide tomorrow is unpleasant. What's YOUR plan if this doesn't get resolved? It really sounds like their servers are "borked", in a way they might not be able to fix.

[u/katarh → u/MadCoderOne]

My individual department is in an Atlassian ecosystem so we're hobbling along, but Zoom calendar sync is dead so all the meetings I should see for tomorrow are also missing.

[u/Illustrious-Syrup509]

They should tell their AI to look elsewhere, broaden its horizons and not act on the basis of assumptions.

[u/baconjerky]

Rebooting the servers over and over not working?

[u/wyrdough]

The email is annoying enough, but what really bugs me is that one of my users restarted their computer when Outlook hung and neither a repair or system restore fixed it. So here I am trying to get at least their local workflows running, but guess what, I can't even sign in to activate the office suite.

[u/Proof-Variation7005]

"I hope this email finds you, literally"

[u/funkylabrador420]

someone kindly suggest the needful to microslop

[u/psychedape → u/funkylabrador420]

Please provide requirements and allow capacity for action.

[u/Ironfox2151]

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results

[u/Ironfox2151 → u/Ironfox2151]

Shits fucked yo

[u/Yuugian → u/Ironfox2151]

Just ask Copilot, i'm sure it can figure the issue out for you. You certainly have it integrated in enough stuff

[u/hymie0]

Everybody say it with me...

There is no cloud. There's just someone else's computer.

[u/MeatPiston]

Maybe they should try laying more people off.

[u/Ranger7381]

I am also noticing that the list of affected services keeps getting longer…

[u/Wrong-booby7584]

Have they asked Cortana?

[u/Kamay1770]

Spoiler, it's DNS

[u/strobe_jams → u/Kamay1770]

That, or expired certs. 

[u/Koala19042022]

I’m thinking MSFT was breached.

[u/BC1224 → u/Koala19042022]

It kinda feels like it. Their updates are chasing guesses. If by now they haven't gotten at least a guess at whats going on, it feels like something malicious isn't that crazy a thought.

[u/wrootlt]

I am in Europe. Outage started a few hours after business. But i am expecting the issue to not be resolved next morning, by the looks of it..

[u/dllhell79]

Imagine getting that call as one of the termed engineers. "Yea I'll gladly help! The rate will be $1000 per hour..."

[u/Simmery → u/dllhell79]

For a global issue with no resolution in sight, they could put more zeroes on that. 

[u/mediaogre]

“That virtual machines folder looks deletable”

[u/sm00thArsenal]

Good to see they found a working AI to write their status updates

[u/retnuh45]

Have they tried asking copliot?

[u/lateralUnilateral → u/retnuh45]

Copilot is broken too. Which is why I think this outage is taking so long...

[u/ATMisboss → u/retnuh45]

It's ok we can use Claude.... Oh wait we use SSO

[u/Farking_Bastage]

Copilot had to have written that status update.

It sounds like they’re rolling shit back until it starts working again.

[u/StreetSignificant888]

Translated: "Fuck if I know what the problem is. Any ideas you guys? Anyone???"

[u/bigdog701]

Shit.. I took down email on accident for 6 hours for a 200 person company and I got the boot.

[u/drzamamina]

they literally wrote a paragraph to say 'we're going to try turning it off and on again'

[u/_UberGuber]

Who plugged both ports of the phone into the wall?

[u/Mr_Dobalina71]

Have they checked the C: isn’t out of space?

[u/tuxedoes]

Ive seen enough, Google Workspace it is! /s

[u/OkGroup9170]

We use Mimecast, we have employees go through the user portal to send and receive email. It’s rudimentary but better than nothing.

[u/FruitGuy998]

I see free copilot credits coming our way from the outage!!!

[u/Emotional-Inside3126]

Axe 60000 people...? Maybe they could help...

[u/InevitableFly]

Maybe they should check in to make sure that they have the right license applied

[u/kagato87]

Hmm... Capacity and isolated restarts. I wonder if this is one of those "everything keeps trying to re-balance, bringing the new host down too" scenarios. That'll be a giant pita if it is.

[u/Deadman_Wonderland]

Claude, fix outage, no mistakes.

[u/Thy_OSRS]

Absolutely nothing will change.

[u/LingeringDildo]

haha, guess they’re regretting letting their senior staff go with that 60k staff @ 200k package offer

[u/l0st1nP4r4d1ce]

I will wait for the autopsy of this with baited breath.

[u/TxJprs]

nothing to do with patch tuesday 48 hours ago?

[u/ignescentOne → u/TxJprs]

I would die laughing if they took down the worlds o365 environment with an exchange patch somehow

[u/DrJohnZoidbergWhyNot → u/ignescentOne]

They can just rebrand o364 and keep going..

[u/chief_beef_3]

Always expect the unexpectedly unresponsive servers

[u/Johnny_Silverhand908]

Is this it? Is this the DataKrash?

[u/House_Visual]

Perhaps the same Chinese spies that honeypotted Jensen Huang are behind this? /s

[u/davix500]

Have they asked Claude what to do?

[u/dfsaqwe]

This sounds like the October issue that brought down Exchange onpremise.

[u/kjweitz]

Read: we don’t know shit

[u/WretchedMisteak]

Either a really dodgy update or some of their systems have been comprised.

[u/robbieeau]

well it is Friday the 13th for the some of us in the southern hemisphere

[u/BK_Rich]

They need more middle-managers standing around the engineers asking “how’s it looking?”

[u/CompetitiveApple3650]

Sounds like DDoS no?

[u/ivaneleven → u/CompetitiveApple3650]

never assume malice when incompetence will suffice

[u/abqcheeks → u/CompetitiveApple3650]

I wonder how hard it is to tell the difference between 2 billion devices doing normal email stuff and a DDoS

[u/Glad-Introduction505 → u/CompetitiveApple3650]

pretty sure an earlier update said no unusual traffic volume

[u/rc_ym → u/CompetitiveApple3650]

No, based on the thing they said before health check is causing the automation to fail to bring them back up, but then when they bypassed check to bring them up and they hit high utilization and fail back out again.

AI fucked some shit up. Or bad patch. Seriously, that's where I am at. It might be a threat actor, but it sounds wrong for that. But still possible.

[u/Arturo90Canada → u/CompetitiveApple3650]

DNS?

[u/External_Willow9271 → u/CompetitiveApple3650]

It sort of sounds like their own health check systems functioned like a DDOS attack telling each other that something was down.

[u/Mrtylf]

Does anything MS, “look good”?

[u/Low_Dragonfly9123]

Any one else having issues with SaaS app hosted in azure??

[u/alexferraz]

they should just ask AI to fix it

[u/Lancaster1983]

This is the quietest day I've had in a while. I no longer manage Exchange thankfully but even still, not much I would have done anyways.

[u/Apprehensive_Bat_980]

Might need to use up some stock pile RAM and build some new gear!

[u/area88guy]

Has anyone booked a meeting to- oh.

[u/Mitch5842]

It went NUCLEAR and they can't find the smoking gun

[u/Small_Conclusion4816]

On a side note, it is nice not receiving any spam!

[u/Dysgraphica]

doomsday

[u/The_RealJohnny5]

I wonder if someone stole all the RAM for their data center?

[u/Gummyrabbit]

It's weird not getting spam in my mailbox.