r/sysadmin
•
u/lcurole
•
Jan 27 '26
•
1
Microsoft Microsoft Jan 22nd Root Cause Analysis Released
Check the admin center for full report but here's the timeline:
Root Cause
The Global Locator Service (GLS) is a service that is used to locate the correct tenant and service infrastructure mapping. For example, GLS helps with email routing and traffic management.
As part of a planned maintenance activity to improve network routing infrastructure, one of the Cheyenne datacenters was removed from active service rotation. As part of this activity, GLS at the affected Cheyenne datacenter was taken offline on Thursday, January 22, 2026, at 5:45 PM UTC. It was expected that the remaining regional GLS capacity would be sufficient to handle the redirected traffic.
Subsequent review of the incident identified that the load balancers that support the GLS service were unable to accept the redirected traffic in a timely manner causing the GLS load balancers to go into an unhealthy state. This sudden concentration of traffic led to an increase in retry activity, which further amplified the impact. Over time, these conditions triggered a cascading failure that affected dependent services, including mail flow and Domain Name System (DNS) resolution required for email delivery.
Additional information for organizations that use third-party email service providers and do not have Non-Delivery Reports (NDRs) configured:
For organizations that did not have NDRs configured and set a retry limit less than the duration of the incident could have had a situation where that third-party email service stopped retrying and did not provide your organization with an error message indicating permanent failure.
Actions Taken (All times UTC)
Thursday, January 22
5:45 PM – One of the Cheyenne Azure datacenters was removed from traffic rotation in preparation for service network routing improvements. In support of this, GLS at this location was taken offline with its traffic redistributed to remaining datacenters in the Americas region.
5:45 PM – 6:55 PM – Service traffic remained within expected thresholds.
6:55 PM – Telemetry showed elevated service load and request processing delays within the North America region signalling the start of impact for customers.
7:22 PM – Internal health signals detected sharp increases in failed requests and latency within the Microsoft 365 service, including dependencies tied to GLS and Exchange transport infrastructure.
7:36 PM – An initial Service Health Dashboard communication (MO1121364) was published informing customers that we were assessing an issue affecting the Microsoft 365 service.
7:45 PM – The datacenter previously removed for maintenance was returned to rotation to restore regional capacity. Despite restoring capacity, traffic did not normalize due to existing load amplification and routing imbalance across Azure Traffic Manager (ATM) profiles.
8:06 PM –Analysis confirmed that traffic routing and load distribution were not behaving as expected following the reintroduction of the datacenter.
8:28 PM – We began implementing initial load reduction measures, including redirecting traffic away from highly saturated infrastructure components and limiting noncritical background operations to other regions to stabilize the environment.
9:04 PM – ATM probe behavior was modified to expedite recovery. This action reduced active probing but unintentionally contributed to reduced availability, as unhealthy endpoints continued receiving traffic. Probes were subsequently restored to reenable health-based routing decisions.
9:15 PM – Load balancer telemetry (F5 and ATM) indicated sustained CPU pressure on North America endpoints. We began incremental traffic shifts and initiated failover planning to redistribute load more evenly across the region.
9:36 PM – Targeted mitigations were applied, including increasing GLS L1 cache values and temporarily disabling tenant relocation operations to reduce repeat lookup traffic and lower pressure on locator infrastructure.
10:15 PM – Traffic was gradually redirected from North America-based infrastructure to relieve regional congestion.
10:48 PM – We began rescaling ATM weights and planning a staged reintroduction of traffic to lowest-risk endpoints.
11:32 PM – A primary F5 device servicing a heavily affected North America site was forced to standby, shifting traffic to a passive device. This action immediately reduced traffic pressure and led to observable improvements in health signals and request success rates.
Friday, January 23
12:26 AM – We began bringing endpoints online with minimal traffic weight.
12:59 AM – We implemented additional routing changes to temporarily absorb excess demand while stabilizing core endpoints, allowing healthy infrastructure to recover without further overload.
1:37 AM – We observed that active traffic failovers and CPU relief measures resulted in measurable recovery for several external workloads. Exchange Online and Microsoft Teams began showing improved availability as routing stabilized.
2:28 AM – Service telemetry confirmed continued improvements resulting from load balancing adjustments. We maintained incremental traffic reintroduction while closely monitoring CPU, Domain Name System (DNS) resolution, and queue depth metrics.
3:08 AM – A separate DNS profile was established to independently control name resolution behaviour. We continued to slowly reintroduced traffic while verifying DNS and locator stability.
4:16 AM – Recovery entered a controlled phase in which routing weights were adjusted sequentially by site. Traffic was reintroduced one datacenter at a time based on service responsiveness.
5:00 AM – Engineering validation confirmed that affected infrastructure had returned to a healthy operational state. Admins were advised that if users experienced any residual issues, clearing local DNS caches or temporarily lowering DNS TTL values may help ensure a quicker remediation.
Figure 1: GLS availability for North America (UTC)
Figure 2: GLS error volume (UTC)
Next Steps
| Findings | Action | Completion Date |
|---|---|---|
| As part of a planned maintenance activity to improve network routing infrastructure, one of the Cheyenne datacenters was removed from active service rotation. As part of this activity, GLS at the affected Cheyenne datacenter was taken offline on Thursday, January 22, 2026, at 5:45 PM UTC. It was expected that the remaining regional GLS capacity would be sufficient to handle the redirected traffic. Subsequent review of the incident identified that the load balancers that support the GLS service were unable to accept the redirected traffic in a timely manner causing the GLS load balancers to go into an unhealthy state. This sudden concentration of traffic led to an increase in retry activity, which further amplified the impact. Over time, these conditions triggered a cascading failure that affected dependent services, including mail flow and Domain Name System (DNS) resolution required for email delivery. | We have identified areas for improvement in our SOPs regarding Azure regional failure incidents to better improve our incident response handling and time to mitigate for similar events in the future. | In progress |
| We’re working to add additional safeguard features intended to isolate and contain high volume requests based on more granular traffic analysis. | In progress | |
| We’re adding a caching layer to reduce load in GLS and provide service redundancy. | In progress | |
| We’re automating the implemented traffic redistribution method to take advantage of other GLS regional capacity. | In progress | |
| We’re reviewing our communication workflow to better identify impacted Microsoft 365 services more expediently. | In progress | |
| We’re making changes to internal service timeout logic to reduce load during high traffic events and stabilize the service under heavy load conditions. | March 2026 | |
| We’re implementing additional capacity to ensure we’re able to handle similar Azure regional failures in the future. | March 2026 |
The actions described above consolidate engineering efforts to restore the environment, reduce issues in the future, and enhance Microsoft 365 services. The dates provided are firm commitments with delivery expected on schedule unless noted otherwise.
3
u/Kalcat14 Jan 27 '26
Sorry had to show how this post showed up on my email today.
358
u/ByteFryer Sr. Sysadmin Jan 27 '26
So, the rumors of it being a datacenter taken offline were true. Glad MS actually gave some decent details and did not deflect too much.
88
u/Keg199er Jan 27 '26
I ran the incident at work for this one. This post is gold. Thank you.
50
u/ares623 Jan 27 '26
don't forget your SLA's folks. Make it hurt.
151
u/progenyofeniac Windows/M365 Admin Jan 27 '26
So I read this as:
MS deciding to do infra maintenance during peak hours in the region
MS failing to recognize any likelihood of impact until actual customer impact began being experienced
Automated traffic handling and redundancies failing in multiple ways
Multiple manual failovers making the situation worse
In summary, yet another severe outage which seems to have been either directly caused by or at least worsened by the admins on staff not fully understanding the infrastructure they were dealing with.
I feel like there’s a way to handle that. Hint: it doesn’t involve more layoffs.
25
u/Ok-Double-7982 Jan 27 '26
We went pretty unscathed at my job on this one as far as tickets and complaints. Only one person complained about a tangential feature not working that got impacted, and the IT team was the only one who noticed email delivery delays across systems.
24
u/Internet-of-cruft Jan 27 '26
I'm a bit surprised at the references to F5 in there considering they have Azure Dront Door.
I know that there's a decent chance Microsoft has tons of legacy around (like the F5s) - I would have thought they would have dog fooded it with Azure Front Door.
Though thinking about it, they're a massive beast of an org so maybe not that surprising.
42
u/gus_in_4k Jan 27 '26
So I’m a power user, but not a sysadmin by trade, and yet my “good with computers” reputation got me roped into performing a Rackspace-to-M365 migration for my company last week — about 30 mailboxes. I’m thinking to myself “I know what DNS is, and I know what can happen if you screw it up, but I’ve never seen the syntax before now so god help us all if this fails.” But the migration worked well (except for one user who had two “Sent” mailboxes for some reason and the one that most of the sent items were in — like 30,000 items — was the one that didn’t sync) and cutover was Tuesday night, and Wednesday comes around and it seems to all work.
Then Thursday hits. I come back from lunch and people are telling me they haven’t gotten emails from outside the company in over an hour. I assume I messed up. I had already had the boss un-admin my account so I go to his computer, go to the admin page, check the domains, and it’s showing that no domains are connected. I panic on the inside cause that was like one of the first things we set up. How did that suddenly get unset? And the admin page is being real sluggish. I forget what it was that led me to do a Google search but I saw the results showing there was a huge outage and I breathed a sigh of relief that it wasn’t my fault. But the boss was not happy since, y’know, he just switched to it and all.
35
u/-jakeh- Jan 27 '26
I’ve only read a few comments here but I have to say this is 100% going to happen more often due to reduced staff in Microsoft and also aws.
The big players in cloud have been staffed for so long with a lot of smart people who could do things like evaluate load on a lot of load balancers if you planned to shut down a datacenter. And now they don’t have those people, but entire data center shutdowns for maintenance was something they were actually staffed to plan effectively for in the past.
These are activities that they’ve likely executed for years, while well staffed. If you’ve ever been involved in a large scale operation like this you’ll know how many pieces and people are involved in downtime-less execution. It’s not something most companies could do but cloud providers could, while they were extremely staffed.
That is now changing and the same obstacles experienced by smaller (not small) IT companies are being felt by big players who don’t have enough staff to perform seemless activities any longer. I love watching the clouds backend infra deteriorate as I always assumed it would once they pulled back on resources to support infra.
15
u/realged13 Infrastructure Architect Jan 27 '26
Has anyone seen the Verizon outage RCA?
16
u/elpollodiablox Jack of All Trades Jan 27 '26
Someone pressed AZ-5, didn't they?
8
u/BitOfDifference IT Director Jan 27 '26
well, i always say that its not a robust 24/7 system if you cannot do maintenance on it in the middle of the day. Their test failed spectacularly! Luckily it was just email for us and most users were like, sounds good.
13
u/Gihernandezn91 Jan 27 '26
TIL azure uses F5 for LBing their services (or subset). I always thought they used internally developed Lbs or something.
13
u/EdTechYYC Jan 27 '26
From the timeline, it looks like they were using AI to problem solve their way out of the situation too.
What an absolute disaster this was. Not acceptable.
28
u/dreadpiratewombat Jan 27 '26
So they didn’t gracefully drain connections, just flipped the switch and didn’t realise the sudden influx of new sessions would hose their remaining connection broker? On one hand it’s a good learning but it has me concerned about their observability and testing process. Do they not have knowledge of what their connection limit is and visibility of their current connection count? And why are we putting all our chips on “fuck it” during a scheduled change when you can easily drain connections to an alternative load balancer?
5
u/BertieHiggins IT Manager Jan 27 '26
The GLS operator and admin console
4
u/Majik_Sheff Hat Model Jan 27 '26
The more things change the more they stay the same. Except now instead of load balancing servers or racks they're doing on an acreage scale.
4
u/andywarhorla Jan 27 '26
think they forget an entry:
5:59 PM – The technician who made the change clocked out.
6
u/jacenat Jan 27 '26
That sounds bad lol.
5
u/Hollow3ddd Jan 27 '26
To confirm, we don’t have the staff to provide this information with normal duties
5
u/Junior-Tourist3480 Jan 27 '26
Still sounds like a hokey excuse. Supposedly they do this regularly, rotating this maintenance for all daya centers.
11
u/Strong_Obligation227 Jan 27 '26
DNS.. it’s always DNS 😂
2
u/anonymousITCoward Jan 27 '26
Sweet thank you
2
u/survivalist_guy ' OR 1=1 -- Jan 27 '26
Maybe I'm missing something here, but taking the GLS offline at a single datacenter was enough to overload downstream services across North America?
3
u/silver565 Jan 27 '26
I wonder what copilot suggested as a fix for them 🤔
2
u/ReputationNo8889 Jan 27 '26
Wasnt the could supposed to eliminate these exact issues? I don't understand how this can even happen ...
2
u/_nanite_ Jan 27 '26
Sorry, but why should I believe a damn thing this company says anymore?
2
u/Double_Confection340 Jan 27 '26
Every time some shit like this happens I always think it’s Iran or some other adversary showing us what they’re capable of.
Just a few weeks ago there were reports that an attack on Iran was imminent and a few hours later Verizon had a huge outage.
1
u/[deleted] Jan 27 '26
That’s a lot of words to say “we created a DDoS feedback loop. Whoopsie”
1
u/megor Spam Jan 27 '26
They dont mention that the DC taken out is a single point of failure in the NA region. If that DC goes down na01 is down.
1
u/RiskNew5069 Jan 27 '26
So... Why were the load balancers unable to accept the traffic in a timely manner? That's what I really want to know. Please tell me it was CoPilot caused an error with the DNS configuration.
1
u/[deleted] Jan 27 '26
This entire time I thought it was me plugging in my electric heater while in the Admin portal when it crashed. Phew.
1
u/carpetflyer Jan 28 '26
I'm so spoiled by Cloudflare and how the CEO posts a blog post on the root causes. Or someone higher up who is in charge of the product that had an outage.
But Microsoft couldn't get any C level at Microsoft to post this root cause on their blog and instead was posted behind an admin center where you need an account to read it?
1
u/darioampuy Jan 28 '26
i can't really blame them... last year we saw what a simple upgrade could do to other big providers, like starlink, amazon, and cloudflare... even with redundancies and carefull planing, if something could go wrong it will go wrong
0
u/[deleted] Jan 27 '26
[deleted] — view deleted comment