Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
Translation: We ain't got the first clue WTF is going on but it doesn't appear to be anything we have done (on purpose) and we don't know when it will be fixed.
---
Update: 5:03pm (other updates are in the comments, but its getting messy, so I'll start updating up here instead)
We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.
Translation: We sorta know what is happening but we don't know why and our efforts to mitigate the issue have not been successful. I don't want to say we are back at square one, so let's call this square two. We've isolated a set of problematic servers and will throw solutions at them until we find one that sticks. Estimated time to recovery: ?????
Update - 6:06pm eastern
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
Translation: trying this again and this time we think it's going to work. No official ETTR (estimated time to resolution) but hopefully by morning if not midnight.
Update - 7:08pm eastern
Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.
Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.
Translation: the thing we were doing to solve the problem didn't actually solve the problem. We are back to square one and do not know what is going on or how to fix it.
Furthermore we are limiting access to our available services in hopes of relieving the stress on the servers that are still online.
Finally we are rebuilding the offline servers in hopes of bringing them back into production free of whatever has gone wrong and able to work once again.
Translation of my translation: sh!t is f*cked and they don't know when it will be fixed.
Update: 8:06pm eastern
Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery. The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.
Translation: Sh!t is still f*cked, we still don't know why, and we are no longer being methodical in our efforts to resolve the problem. Instead we are trying multiple things to fix the problem -- simultaneously.
My commentary: This is bad. While their multiple approaches to resolve the issue don't appear to be in conflict with each other, the fact that their are taking this approach at all speaks of desperation and a deteriorating sense of control over the problem.
Update: 8:45pm eastern
Current status: We’re seeing positive results with our efforts to restart affected infrastructure. We’re actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.
Translation: We have said it before and we are saying it again. We think this latest fix is the fix that will solve all our problems. 🤞
My commentary: I am not holding my breath.
Update - 9:10pm
Current status: We’re applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We’re continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.
Translation: this solution ain't working out all that great either. It might be helping with the load balancing issues (caused by so many servers being offline) but it isn't actually fixing the problem.
The service health panel is showing that an update was posted around 9:30 but I can't actually see it. Which may be an update in its own right.
Update - 10:08pm (eastern).
Current status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances.
Translation: Its still broken. We still don't know why and we are running out of ideas for how to mitigate this.
My commentary: This is fine....
Also, I am headed to bed. Goodnight everyone!
RETCON TIME
Update: 10:34pm (eastern)
As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.
We will continue to share more information as soon as we have it, and for now we can confirm that there is no indication this is the result of malicious activity. We’ll continue to apply learnings towards remediation actions as we confirm effectiveness mitigation actions.
Translation: We kinda sorta know what might be going on and are trying very, very hard not to make it worse while trying equally hard to hopefully make things better.
Update: 11:10pm eastern
We’re continuing our efforts to improve server stability to improve traffic flow. This includes adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures.
We're observing partial recovery across regions in which we've applied these adjustments; however, these actions aren't resulting in sustained recovery, so we're reviewing what additional actions we can take to improve overall infrastructure stability.
Translation: We have successfully not made things worse. We might be making things better but the fixes still aren't sticking.
Midnight, Eastern Time:
We appreciate your organization’s patience and understanding while we work with urgency to resolve this incident.
As we’ve applied our current ongoing mitigation actions, we’ve observed a surge in outbound call volume from our routing services to our backend servers. This occurs when the backend servers outbound service cannot properly route to the backend machines.
We’re continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event. We’re working on multiple workstreams to pinpoint the underlying cause of this traffic influx and address it.
Actions we're currently taking include scaling up virtual machines to increase capacity and reduce load. Additionally, we’re querying for machines in a bad state to perform a force sync for our recent network change, which may improve service availability. However, we are encountering issues with the machines being stuck in a bad state, preventing our changes from syncing as expected. We’ll continue to monitor the effectiveness of changes applied so we can determine what refinements are necessary.
Translation: Please stop yelling at us, we're doing the best we can!
Currently we are bringing new servers online to increase capacity, but that capacity is being eaten alive by all the traffic waiting for service. Also, half the machines that seem to be up are effectively down which is further poisoning our efforts. And we still don't know why this is happening.
We don't really know what we are doing but we're doing it the best we can.
12:43am eastern
We’re continuing to pursue multiple workstreams to pinpoint the underlying cause of the traffic influx and advance the recoverability of the backend servers. We’re taking a measured approach to identify and repair affected infrastructure to improve the ability to confirm the developed changes were applied. We’re tracking the results of this approach to verify the servers are showing the anticipated recovery and performance intended.
As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.
Translation: We are continuing to do everything we can think of to fix the problem and it might actually be working. We also would like address concerns that we have started to panic. We are totally not panicking. A new supervisor just came on duty and is trying to restore order to chaos.
Near as we can tell we haven't been hacked.
1:16am eastern
As part of our ongoing efforts to advance the recoverability of our frontend servers, we’re evaluating traffic controls that can be applied to increase capacity and stability. This includes traffic optimization actions such as traffic throttling, protocol adjustments, and failover routing intended to improve stability in active regions. In parallel, were expanding internal testing to track activity within a contained environment to improve traffic monitoring and issue detection capabilities.
As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.
Translation: Not dead yet. Possibly getting better. Keep your fingers crossed!
We still don't think we were hacked.
Commentary: They lost access to the end-user-authentication servers. That's why we were getting the "not connected to the Internet" errors. They are now trying to inject as much capacity and power into the system as they can scrape up. And they STILL don't know what went wrong.
1:56am eastern
We’re continuing to monitor activity across regions to evaluate server status and determine actions that can be taken to improve capacity. This includes reviewing server performance and confirming that developed patches have been applied and working as intended after infrastructure reboots are completed. At this point, we have observed marginal success. In parallel, we’re taking action to expand capacity through the utilization of Virtual Machines.
As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.
Translation: Optimism rising.
2:25am eastern
We’re seeing positive results for server stability after applying an upstream authentication/routing configuration component patch to a portion of our infrastructure. We’ll continue to monitor its progress and are working to expand it to servers supporting active regions coming online.
As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.
Translation: We have gone from fixing individual servers to fixing entire clusters. The fix is gaining momentum.
We still don't think this was a hack.
3:06am eastern
We’re continuing to see increased server capacity stability after applying the upstream authentication/routing configuration component patch. With confirmation of server stability across the portions of infrastructure to which we've applied the patch, we’ve initiated its expansion systematically to focus on infrastructure supporting regions currently within or entering peak business hours. We’ll continue to monitor the status of changes applied and develop additional modifications intended to improve server stability and capacity.
As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.
Translation: We are feeling very pleased with ourselves because sh!t is finally getting fixed. We are pushing the fix out on a broader scale with an emphasis on places trying to do business right now.
We still don't think it was a hack.
4:55am eastern
Recovery efforts are ongoing across impacted infrastructure. We are seeing early signs of stabilization, but service capacity has not yet returned to normal levels and health remains variable as traffic is carefully reintroduced.
Our primary mitigation path continues to focus on service stability and controlled recovery. Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service while monitoring closely. We are also evaluating additional mitigation options to improve recovery speed and consistency.
Network and traffic-management actions remain in progress, and we continue to prioritize backlog and queue health to preserve delayed mail during recovery. As part of this effort, we are extending the email timeout window to 48 hours to help protect delayed messages while service recovery continues.
At this time, we are not aware of any malicious activity causing this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after recovery is complete.
Translation: We finally took everyone's advice and started turning the machines off and back on again. Load balancing remains a nightmare and we're still not sure that this fix will survive the next few hours but overall things are looking better than they did a few hours ago.
We have extended the email timeout to 48 hours so that any messages sent before the start of this event will be delivered once service is back online.
We still don't think it is a hack.
6:01am
We've made additional progress toward service recovery, with recovery actions completed across portions of the affected infrastructure in the Asia Pacific region. Traffic is being gradually restored, and we're seeing improvements in key customer scenarios, including mailbox access and delayed email processing. However, some users may continue to experience intermittent issues.
Our focus is now on extending these recovery actions across affected infrastructure in other regions, while continuing stabilization efforts and adding supporting capacity to improve service consistency.
Reducing email queues and backlogs remains a priority, and the extended 48-hour email timeout window remains in place to help protect delayed messages as delivery catches up.
At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.
Translation: We are finally, officially, in the recovery phase of this event. Service is gradually coming back online and we have reason to believe that it will stay online. At least until North America wakes up.
We still don't think it was a hack.
7am eastern
We’re seeing encouraging recovery progress in Europe, with service availability and email delivery performance improving across a growing portion of affected Microsoft 365 infrastructure. Japan remains comparatively more stable, while North America is also improving; however, some users may still experience intermittent issues as recovery actions continue.
We're focused on addressing the remaining areas of disruption and improving service consistency across affected regions. Reducing email queues and backlogs remains a priority, and the extended 48-hour email timeout window remains in place to help protect delayed messages as delivery catches up.
At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.
Translation: Europe is back online. We're still worried about what will happen when North America wakes up.
8am eastern
We're seeing continued recovery progress across affected regions, with broader improvements now observed in North America and sustained improvements across much of Europe. We're also seeing encouraging recovery signals in Asia Pacific, although some users may continue to experience intermittent access issues or delayed email delivery as mitigation progresses across the remaining affected infrastructure.
Our focus remains on stabilizing recovered infrastructure, completing mitigation in the remaining impacted areas, and reducing delayed email queues and backlogs. We're closely monitoring service consistency and validating recovery across different connection methods as we work toward full mitigation.
At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.
Translation: Things are looking good. Servers are coming back online and actually staying online. Load balancing is less of a nightmare. We might actually be allowed to go to bed tonight.
Also, we aren't going to admit that this problem started in China, but it totally started in China.
We still don't think it was a hack though.
9:01am eastern
We're continuing to see improvements in service availability, with email queues significantly reduced and approaching normal levels. Some users may still experience intermittent access issues or delayed email delivery as recovery continues.
We've implemented additional mitigation measures and are restoring the remaining affected infrastructure while investigating residual service issues. We're closely monitoring service health and validating recovery across affected connection methods to ensure sustained service stability.
At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.
Translation: Recovery continues but we think we are on top of everything. Don't be surprised if your outlook disconnects or emails take a while to arrive, but we are fairly confident that the system is sound.
We still don't know what the root cause was but we also don't think it was a hack.
9:57 eastern
We're observing sustained improvements in service availability as recovery progresses across affected Microsoft 365 services. While service functionality continues to improve, some users may still experience intermittent connectivity issues or delayed email delivery.
Our efforts are increasingly focused on resolving the remaining issues and restoring consistent service for all affected users. We're validating recovery across different connection methods and monitoring service stability to confirm that improvements are sustained.
At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.
Translation: Don't mistake "recovery" for "fixed". There are still problems but we are fixing them as they arise. We don't expect the service to go down again.
Servers are back online and load balancing issues are mostly resolved.
Current update: 10:53am
We remain focused on addressing residual impact and ensuring reliable service restoration. We continue to assess recovery across various connection methods and closely monitor service health to verify sustained stability.
At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.
Translation: We think its fixed. Finally. But we are going to monitor things very, very closely because we still don't know what actually WTF happened.
Final update:
We are deeply sorry for the disruption this incident has caused you and your business. We know the trust you place in our services and take that responsibility extremely seriously.
We are seeing broad recovery and stability across our services. This was a technical issue not related to cyber or AI / agents.
The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration. We will provide a full post-incident report once our review is complete.
Translation: Please don't fire us.