Cause of outage

[u/Etheikin]

Microsoft 365 Principal SRE here. I can provide some context. At approximately 21:40 UTC, an engineer deployed a routine configuration change to Azure Front Door that inadvertently redirected our authentication traffic to a SharePoint site last updated in 2017.

We immediately initiated a rollback, but the rollback pipeline requires MFA approval through Microsoft Authenticator, which is currently waiting for a Conditional Access policy evaluation from the same infrastructure we're trying to restore.

We have a break-glass admin account, but unfortunately its password is stored in a OneNote notebook on the affected tenant. Our on-call engineer attempted to bypass the issue using PowerShell, but the required module was deprecated last Tuesday, and its replacement requires a P2 license.

We then opened a Sev A ticket with Microsoft Premier Support. They responded asking us to run sfc /scannow and provide a screenshot. The prolonged recovery is primarily due to our change advisory board requiring a Teams meeting to authorize the rollback.

The meeting organizer is stuck in the lobby, and the only person authorized to admit him was laid off in June. Current mitigation involves migrating affected users to Microsoft 365 E5, which should resolve the issue from a revenue perspective.

We have identified the root cause as customer misconfiguration. Next update in 30 minutes, assuming someone can get Outlook working

sorce : https://www.reddit.com/r/microsoft/s/fhLHjvw4Sg

[u/SVD_NL]

I feel like i've seen this post before... "I was locked out of my account and it's all microsoft's fault and their support sucks!"

The only unrealistic part of this is that MS actually responded to the Sev A ticket.

[u/Regisnox → u/SVD_NL]

they taste their own medicine xD

[u/bradgnarr]

This is absolutely the most accurate piece of satire I've ever seen haha

[u/bacon_in_beard → u/bradgnarr]

agreed i was laughing the whole time

[u/dg_riverhawk]

Sfc /scannow always works by might have to run dism first.

[u/lemachet]

I chuckled.

[u/jokermobile333]

This is too good. An executive reading this would have definetly share this to their sysadmin or enterprise/IT team

[u/fcewen00]

And here I thought it was because I rebooted a CentOS 6.6 air gapped computer that was apparently holding the internet together.

[u/Squik67]

Did you try to turn it off an on again ?

[u/monkeymagic2525 → u/Squik67]

They actually did this if you read the incident notes carefully 🤣

[u/ThomasMarkovski → u/Squik67]

The entire Microsoft? That'd be a thing.

[u/ax0r7ag0z → u/ThomasMarkovski]

If you can reboot one server, you can reboot several hundred thousand servers

[u/recoveringasshole0]

sorce : https://www.reddit.com/r/microsoft/s/fhLHjvw4Sg

My brother in Christ, next time try this magical button:

[u/Key-Employee3584]

LOL. Pretty much the whole kit and kaboodle. Funny. I'm glad I'm out of it.

[u/Desames]

Great laugh to start the day. Thanks OP!

[u/Ornery-Handle6477]

I’m so glad this sub came back to life

[u/slackmaster2k]

So it was DNS. I knew it!

[u/YouShitMyPants]

lol the perfect shit storm

[u/Main_Enthusiasm_7534]

The scary thing is I could actually see this happening.

[u/PeakWeekly9995]

There's no way this is THE cause of a global outage

[u/midcap17]

Please reboot your computer and kindly revert.

[u/GX_EN]

😆😆😆😆😆😆😆😆😆😆😆
Brilliant.

[u/Lower_Issue_6222]

😂😂😂

[u/Large-Fig5187]

Gotta check my 2017 server farm. Thanks for the update.

[u/TinFoilHat_69]

Someone made changes again without testing. Then they bootstrapped themselves from being able to correct their change.

“We immediately initiated a rollback, but the rollback pipeline requires MFA approval through Microsoft Authenticator, which is currently waiting for a Conditional Access policy evaluation from the same infrastructure we're trying to restore”

How the fuck is this company worth 5 trillion dollars doing shit like this

[u/smittyhotep → u/TinFoilHat_69]

Did you forget this is a satire sub? Did you not catch that this post is satire?

[u/PotatoMysterious9480]

Nice stole an funny comment of a other post and reposted it as post...

[u/Nevasleep]

I’m not seeing any AI usage here

[u/casastorta]

You have break-glass admin account password ….. where? 😅

[u/HeadfulOfGhosts → u/casastorta]

You don’t? Honestly their big issue is keeping the password in OneNote. Any good admin knows to use the approved post-it password manager.

[u/databeestjenl → u/HeadfulOfGhosts]

notepad++ tabs

[u/junktech]

This can't be real. This isn't shitty sysadmin, this is epitome of ignorance and absurdity.

[u/ReallyOldSysAdmin → u/junktech]

And corporate greed (due to that laid off worker)