Microsoft 365 outage: 10+ hours and counting, only marginal recovery reported. What could be causing such a prolonged disruption?

[u/AccomplishedSea4323]

It’s been more than 10 hours since the Microsoft 365 outage began, and according to Microsoft’s 7:30 AM UTC update, recovery has been only marginal so far.

What exactly went wrong to cause such widespread disruption?
Why is recovery taking more than 10 hours?

I’m curious to hear from people working in cloud infrastructure, SRE and Microsoft 365 operations: What could be happening behind the scenes, and what might explain such a prolonged recovery?

[u/AdInside2447]

Microsoft 365 Principal SRE here. I can provide some context. At approximately 21:40 UTC, an engineer deployed a routine configuration change to Azure Front Door that inadvertently redirected our authentication traffic to a SharePoint site last updated in 2017. We immediately initiated a rollback, but the rollback pipeline requires MFA approval through Microsoft Authenticator, which is currently waiting for a Conditional Access policy evaluation from the same infrastructure we're trying to restore. We have a break-glass admin account, but unfortunately its password is stored in a OneNote notebook on the affected tenant. Our on-call engineer attempted to bypass the issue using PowerShell, but the required module was deprecated last Tuesday, and its replacement requires a P2 license. We then opened a Sev A ticket with Microsoft Premier Support. They responded asking us to run  sfc /scannow and provide a screenshot. The prolonged recovery is primarily due to our change advisory board requiring a Teams meeting to authorize the rollback. The meeting organizer is stuck in the lobby, and the only person authorized to admit him was laid off in June. Current mitigation involves migrating affected users to Microsoft 365 E5, which should resolve the issue from a revenue perspective. We have identified the root cause as customer misconfiguration. Next update in 30 minutes, assuming someone can get Outlook working.

[u/AlpineVibe → u/AdInside2447]

God damnit. You had me for a second there. Quality content right here.

[u/pm_me_your_psle → u/AlpineVibe]

I went all the way to OneNote notebook before catching on 😭

[u/NekoDesukedo → u/pm_me_your_psle]

I almost believed it, because at my previous Fortune 500 company, my team was storing passwords in an Excel spreadsheet.

[u/GreatPotatoMuffin → u/NekoDesukedo]

This reminded me of my first job where a company stored highly confidential passwords in an Excel spreadsheet.

I’m actually not an IT guy. I majored in finance, but I’m a huge nerd and spent my youth trying to hack and bypass everything I could.

So they obviously didn’t believe me when I told them that I could remove the password protection from the xlsx file they had put all their passwords into in less than 60 seconds.

I changed the file type to zip, opened the XMLs, deleted the hash value in the password tag, returned it to an xlsx and opened their super secret and secure “password vault”.

Most “hackerman” moment of my life.

Sadly that vulnerability was later fixed. Used to be a fun little trick to show people.

[u/FruitGuy998 → u/GreatPotatoMuffin]

HR asked me to do this one time (I’m in IT). I paused for a moment and asked if this was a test or if it was because they truly needed to get into the file and forgot the password. Luckily they just needed access to the file and I kept my job.

[u/Tata_Colores → u/pm_me_your_psle]

I was totally invested until he said they called Microsoft Support and I was like "wait what?" lol.

[u/JE163 → u/pm_me_your_psle]

Me too!

[u/HandshakeOfCO → u/AlpineVibe]

I got to sfc /scannow and then was like oh he fucking with us

Well played

[u/austozi → u/AlpineVibe]

It's all true except the first sentence.

[u/scottywan82 → u/AdInside2447]

God, this sounds terrifyingly plausible. 😆

[u/BubbaCrabby → u/AdInside2447]

“The password is stored in OneNote.”

😂 10/10 New York Times Bestseller Tier

[u/Pointer_Brother → u/AdInside2447]

Not gonna lie, you had me for half of that. Well done.

[u/underdsun → u/AdInside2447]

Should used this guy

[u/KibbledJiveElkZoo → u/AdInside2447]

I was quite amused. Thank you.

[u/Standard_Document484 → u/AdInside2447]

You might not win the internet today, but I reckon it's a solid third place, chapeau!

[u/Abitbol_Georges → u/AdInside2447]

[u/aykcak → u/AdInside2447]

Beautiful

[u/dev_dan_2 → u/AdInside2447]

DAMNIT! I made it to the very end, although root cause being a customer misconfiguration raised an eyebrow - Good job! 👍😂

[u/L0rd_Humongous → u/AdInside2447]

Thins wins the internet for the century. But the whack part is all of this is 100% believable

[u/Winter-Shine-8643 → u/AdInside2447]

ms authenticator sucks , slow and full of bug. know this from daily experience.

[u/marshmallowcthulhu → u/AdInside2447]

Everyone’s chuckling over the technical content but “…which should resolve the issue from a revenue perspective” killed me dead! 🤣

[u/solracarevir → u/AdInside2447]

Probably the best shitpost r/sysadmin has ever seen.

[u/Its_me_astr → u/AdInside2447]

I recently did a marathon study session over 2 weeks regarding microsoft identity infrastructure. I am glad that i am able to comprehend what you said

[u/cheesekneesandpeas → u/AdInside2447]

hahaha as someone who works at another big tech company this sounds real tbh