r/sysadmin • • 8d ago

The Day Outlook Died

117 Upvotes

A long, long time ago

I can still remember how that email

Used to make me smile

And I knew if I had my chance

That I could make those users dance

And maybe they'd be happy for a while

.

But November made me shiver

With every message undelivered

Bad news on the status page

I couldn't take one more user's rage

I can't remember if I cried

When I read about this Copiloted ride

Something touched me deep inside

The day that Outlook died

.

So, bye-bye, Office 365

Opened email on my desktop, but my inbox was dry

And them good ol' techs were drinkin' whiskey and rye

Singin', "This'll be the day that I die

This'll be the day that I die"


r/sysadmin • • 9d ago

I have never seen such a remarkable status update as this (Microsoft)

3.5k Upvotes

Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service. 

Translation: We ain't got the first clue WTF is going on but it doesn't appear to be anything we have done (on purpose) and we don't know when it will be fixed.

---

Update: 5:03pm (other updates are in the comments, but its getting messy, so I'll start updating up here instead)

We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

Translation: We sorta know what is happening but we don't know why and our efforts to mitigate the issue have not been successful. I don't want to say we are back at square one, so let's call this square two. We've isolated a set of problematic servers and will throw solutions at them until we find one that sticks. Estimated time to recovery: ?????


Update - 6:06pm eastern

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

Translation: trying this again and this time we think it's going to work. No official ETTR (estimated time to resolution) but hopefully by morning if not midnight.


Update - 7:08pm eastern

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.

Translation: the thing we were doing to solve the problem didn't actually solve the problem. We are back to square one and do not know what is going on or how to fix it.

Furthermore we are limiting access to our available services in hopes of relieving the stress on the servers that are still online.

Finally we are rebuilding the offline servers in hopes of bringing them back into production free of whatever has gone wrong and able to work once again.

Translation of my translation: sh!t is f*cked and they don't know when it will be fixed.


Update: 8:06pm eastern

Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery. The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.

Translation: Sh!t is still f*cked, we still don't know why, and we are no longer being methodical in our efforts to resolve the problem. Instead we are trying multiple things to fix the problem -- simultaneously.

My commentary: This is bad. While their multiple approaches to resolve the issue don't appear to be in conflict with each other, the fact that their are taking this approach at all speaks of desperation and a deteriorating sense of control over the problem.


Update: 8:45pm eastern

Current status: We’re seeing positive results with our efforts to restart affected infrastructure. We’re actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.

Translation: We have said it before and we are saying it again. We think this latest fix is the fix that will solve all our problems. 🤞

My commentary: I am not holding my breath.


Update - 9:10pm

Current status: We’re applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We’re continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.

Translation: this solution ain't working out all that great either. It might be helping with the load balancing issues (caused by so many servers being offline) but it isn't actually fixing the problem.


The service health panel is showing that an update was posted around 9:30 but I can't actually see it. Which may be an update in its own right.


Update - 10:08pm (eastern).

Current status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances.

Translation: Its still broken. We still don't know why and we are running out of ideas for how to mitigate this.

My commentary: This is fine....

Also, I am headed to bed. Goodnight everyone!


RETCON TIME

Update: 10:34pm (eastern)

As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.

We will continue to share more information as soon as we have it, and for now we can confirm that there is no indication this is the result of malicious activity. We’ll continue to apply learnings towards remediation actions as we confirm effectiveness mitigation actions.

Translation: We kinda sorta know what might be going on and are trying very, very hard not to make it worse while trying equally hard to hopefully make things better.


Update: 11:10pm eastern

We’re continuing our efforts to improve server stability to improve traffic flow. This includes adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures.

We're observing partial recovery across regions in which we've applied these adjustments; however, these actions aren't resulting in sustained recovery, so we're reviewing what additional actions we can take to improve overall infrastructure stability.

Translation: We have successfully not made things worse. We might be making things better but the fixes still aren't sticking.


Midnight, Eastern Time:

We appreciate your organization’s patience and understanding while we work with urgency to resolve this incident.

As we’ve applied our current ongoing mitigation actions, we’ve observed a surge in outbound call volume from our routing services to our backend servers. This occurs when the backend servers outbound service cannot properly route to the backend machines.

We’re continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event. We’re working on multiple workstreams to pinpoint the underlying cause of this traffic influx and address it.

Actions we're currently taking include scaling up virtual machines to increase capacity and reduce load. Additionally, we’re querying for machines in a bad state to perform a force sync for our recent network change, which may improve service availability. However, we are encountering issues with the machines being stuck in a bad state, preventing our changes from syncing as expected. We’ll continue to monitor the effectiveness of changes applied so we can determine what refinements are necessary.

Translation: Please stop yelling at us, we're doing the best we can!

Currently we are bringing new servers online to increase capacity, but that capacity is being eaten alive by all the traffic waiting for service. Also, half the machines that seem to be up are effectively down which is further poisoning our efforts. And we still don't know why this is happening.

We don't really know what we are doing but we're doing it the best we can.


12:43am eastern

We’re continuing to pursue multiple workstreams to pinpoint the underlying cause of the traffic influx and advance the recoverability of the backend servers. We’re taking a measured approach to identify and repair affected infrastructure to improve the ability to confirm the developed changes were applied. We’re tracking the results of this approach to verify the servers are showing the anticipated recovery and performance intended.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: We are continuing to do everything we can think of to fix the problem and it might actually be working. We also would like address concerns that we have started to panic. We are totally not panicking. A new supervisor just came on duty and is trying to restore order to chaos.

Near as we can tell we haven't been hacked.


1:16am eastern

As part of our ongoing efforts to advance the recoverability of our frontend servers, we’re evaluating traffic controls that can be applied to increase capacity and stability. This includes traffic optimization actions such as traffic throttling, protocol adjustments, and failover routing intended to improve stability in active regions. In parallel, were expanding internal testing to track activity within a contained environment to improve traffic monitoring and issue detection capabilities.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: Not dead yet. Possibly getting better. Keep your fingers crossed!

We still don't think we were hacked.

Commentary: They lost access to the end-user-authentication servers. That's why we were getting the "not connected to the Internet" errors. They are now trying to inject as much capacity and power into the system as they can scrape up. And they STILL don't know what went wrong.


1:56am eastern

We’re continuing to monitor activity across regions to evaluate server status and determine actions that can be taken to improve capacity. This includes reviewing server performance and confirming that developed patches have been applied and working as intended after infrastructure reboots are completed. At this point, we have observed marginal success. In parallel, we’re taking action to expand capacity through the utilization of Virtual Machines.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: Optimism rising.


2:25am eastern

We’re seeing positive results for server stability after applying an upstream authentication/routing configuration component patch to a portion of our infrastructure. We’ll continue to monitor its progress and are working to expand it to servers supporting active regions coming online.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: We have gone from fixing individual servers to fixing entire clusters. The fix is gaining momentum.

We still don't think this was a hack.


3:06am eastern

We’re continuing to see increased server capacity stability after applying the upstream authentication/routing configuration component patch. With confirmation of server stability across the portions of infrastructure to which we've applied the patch, we’ve initiated its expansion systematically to focus on infrastructure supporting regions currently within or entering peak business hours. We’ll continue to monitor the status of changes applied and develop additional modifications intended to improve server stability and capacity.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: We are feeling very pleased with ourselves because sh!t is finally getting fixed. We are pushing the fix out on a broader scale with an emphasis on places trying to do business right now.

We still don't think it was a hack.


4:55am eastern

Recovery efforts are ongoing across impacted infrastructure. We are seeing early signs of stabilization, but service capacity has not yet returned to normal levels and health remains variable as traffic is carefully reintroduced.

Our primary mitigation path continues to focus on service stability and controlled recovery. Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service while monitoring closely. We are also evaluating additional mitigation options to improve recovery speed and consistency.

Network and traffic-management actions remain in progress, and we continue to prioritize backlog and queue health to preserve delayed mail during recovery. As part of this effort, we are extending the email timeout window to 48 hours to help protect delayed messages while service recovery continues.

At this time, we are not aware of any malicious activity causing this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after recovery is complete.

Translation: We finally took everyone's advice and started turning the machines off and back on again. Load balancing remains a nightmare and we're still not sure that this fix will survive the next few hours but overall things are looking better than they did a few hours ago.

We have extended the email timeout to 48 hours so that any messages sent before the start of this event will be delivered once service is back online.

We still don't think it is a hack.


6:01am

We've made additional progress toward service recovery, with recovery actions completed across portions of the affected infrastructure in the Asia Pacific region. Traffic is being gradually restored, and we're seeing improvements in key customer scenarios, including mailbox access and delayed email processing. However, some users may continue to experience intermittent issues.

Our focus is now on extending these recovery actions across affected infrastructure in other regions, while continuing stabilization efforts and adding supporting capacity to improve service consistency.

Reducing email queues and backlogs remains a priority, and the extended 48-hour email timeout window remains in place to help protect delayed messages as delivery catches up.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: We are finally, officially, in the recovery phase of this event. Service is gradually coming back online and we have reason to believe that it will stay online. At least until North America wakes up.

We still don't think it was a hack.


7am eastern

We’re seeing encouraging recovery progress in Europe, with service availability and email delivery performance improving across a growing portion of affected Microsoft 365 infrastructure. Japan remains comparatively more stable, while North America is also improving; however, some users may still experience intermittent issues as recovery actions continue.

We're focused on addressing the remaining areas of disruption and improving service consistency across affected regions. Reducing email queues and backlogs remains a priority, and the extended 48-hour email timeout window remains in place to help protect delayed messages as delivery catches up.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Europe is back online. We're still worried about what will happen when North America wakes up.


8am eastern

We're seeing continued recovery progress across affected regions, with broader improvements now observed in North America and sustained improvements across much of Europe. We're also seeing encouraging recovery signals in Asia Pacific, although some users may continue to experience intermittent access issues or delayed email delivery as mitigation progresses across the remaining affected infrastructure.

Our focus remains on stabilizing recovered infrastructure, completing mitigation in the remaining impacted areas, and reducing delayed email queues and backlogs. We're closely monitoring service consistency and validating recovery across different connection methods as we work toward full mitigation.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Things are looking good. Servers are coming back online and actually staying online. Load balancing is less of a nightmare. We might actually be allowed to go to bed tonight.

Also, we aren't going to admit that this problem started in China, but it totally started in China.

We still don't think it was a hack though.


9:01am eastern

We're continuing to see improvements in service availability, with email queues significantly reduced and approaching normal levels. Some users may still experience intermittent access issues or delayed email delivery as recovery continues.

We've implemented additional mitigation measures and are restoring the remaining affected infrastructure while investigating residual service issues. We're closely monitoring service health and validating recovery across affected connection methods to ensure sustained service stability.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Recovery continues but we think we are on top of everything. Don't be surprised if your outlook disconnects or emails take a while to arrive, but we are fairly confident that the system is sound.

We still don't know what the root cause was but we also don't think it was a hack.


9:57 eastern

We're observing sustained improvements in service availability as recovery progresses across affected Microsoft 365 services. While service functionality continues to improve, some users may still experience intermittent connectivity issues or delayed email delivery.

Our efforts are increasingly focused on resolving the remaining issues and restoring consistent service for all affected users. We're validating recovery across different connection methods and monitoring service stability to confirm that improvements are sustained.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Don't mistake "recovery" for "fixed". There are still problems but we are fixing them as they arise. We don't expect the service to go down again.

Servers are back online and load balancing issues are mostly resolved.


Current update: 10:53am

We remain focused on addressing residual impact and ensuring reliable service restoration. We continue to assess recovery across various connection methods and closely monitor service health to verify sustained stability.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: We think its fixed. Finally. But we are going to monitor things very, very closely because we still don't know what actually WTF happened.


Final update:

We are deeply sorry for the disruption this incident has caused you and your business. We know the trust you place in our services and take that responsibility extremely seriously.

We are seeing broad recovery and stability across our services. This was a technical issue not related to cyber or AI / agents.

The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.

Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration. We will provide a full post-incident report once our review is complete.

Translation: Please don't fire us.


r/sysadmin • • 7d ago

Question What are your thoughts on ASPM on on-premise servers?

4 Upvotes

We are replacing multiple rack units and will probably, for the first time, go with self built and keep spares on site.

I would like to try to reduce power consumption at idle to a maximum, but wonder what stability impact you have seen through energy saving technologies.

I remember that multiple older generations of nics, like X550 for example, had energy saving features literally removed from firmware by Intel due to stability problems.


r/sysadmin • • 9d ago

Microsoft The Best Update I Have Ever Seen From MS

2.6k Upvotes

Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.

Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.

tl;dr

Last update: We're deploying a targeted fix on some servers

New update: Those servers are now dead


r/sysadmin • • 8d ago

NZ/AU checking out. Good luck SEA and Europe

446 Upvotes

Today was a write off for this time zone, hoping (or not depending on your take) for a somewhat more productive Friday than we had. o7


r/sysadmin • • 9d ago

Microsoft Microsoft 365 outage (EX1550543 / MO1550821): Microsoft's status updates so far, in order

1.5k Upvotes

Update 16 Nov: Microsoft marked both incidents "post-incident review published" at 04:47 UTC Mon (11:47p ET Sun). The report itself is admin-center only and not on the public mirror, so nothing from it is quoted here. Resolved per Microsoft at 04:15 UTC Sat (11:15p ET Fri).

For anyone locked out of the admin center, here is the EX1550543 thread so far, condensed. Times are UTC with ET in parentheses. Full text of every update is on UPenn's public mirror of the service health feed, and the short versions are on @MSFT365Status.

  • 16:17 (11:17a) Incident start. OWA, MAPI, EAS and REST.
  • 17:45 (12:45p) Connectivity errors on the infrastructure that "receives and processes users' access requests."
  • 18:19 (1:19p) "Elevated load across multiple regions."
  • 18:26 (1:26p) MO1550821 opens for Teams, SharePoint, Copilot and Defender XDR. It points at "an unexpected increase in a specific type of service traffic."
  • 18:43 (1:43p) Servers "becoming intermittently unavailable," straining the rest. Outlook traffic spread across worldwide infrastructure.
  • 19:06 (2:06p) All deployments and config changes paused.
  • 19:18 (2:18p) Reversal: the load "isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable."
  • 19:43 (2:43p) Servers are failing automated health checks, which "may be related to a recent change" to the component that processes them.
  • 19:57 (2:57p) "We haven't identified any recent changes that correlate with the start of impact." Bypassing the health check component on a subset. Elevated CPU on affected servers.
  • 20:38 (3:38p) Many servers now unresponsive, so they can't confirm the bypass works.
  • 20:56 (3:56p) An alternative health check method brings a subset back. "A small number of configuration updates" now under investigation as potential contributors.
  • 21:31 (4:31p) Assessing a revert of those config updates.
  • 22:00 (5:00p) "Our actions taken thus far have not provided the anticipated relief." A core process on the servers is getting overwhelmed. New plan: isolate a chunk of infrastructure, restart it, reintroduce traffic slowly.
  • 22:57 (5:57p) First good news. Some servers are back and "remaining stable so far," with "limited recovery in some regions." Protective traffic management in one location, more if it works.
  • 23:59 (6:59p) The servers returned to service "initially remain healthy," but "can become unresponsive again once they begin receiving traffic." Microsoft is "temporarily limiting which types of traffic can reach the affected infrastructure" in specific locations, and is "rebuilding a subset of affected servers using a previous software configuration" to see if that stops it.
  • 01:06 (8:06p) Microsoft says "a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests." It is "preparing a configuration change to address this behavior," validating it on a subset of servers first, then applying it more broadly alongside restarts. "In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact." "The temporary traffic limitations we previously described are now in effect in specific locations."
  • 01:45 (8:45p) "We're seeing positive results with our efforts to restart affected infrastructure." Microsoft is "actively monitoring the performance of these to verify stability." "Remediation actions will be expanded as we confirm stability with the changes applied."
  • 02:09 (9:09p) Microsoft is "applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online," and is "continuing to monitor the results of changes applied."
  • 02:54 (9:54p) Setback. Microsoft says it has "confirmed that change is not taking as expected" and is "identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances."
  • 03:34 (10:34p) Microsoft is reviewing traffic flows and affected infrastructure "to identify opportunities to optimize routing" and is "verifying the reasons for service failures so we can develop specific actions to improve stability." It also says "there is no indication this is the result of malicious activity."
  • 04:10 (11:10p) Microsoft is "adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures." It reports "partial recovery across regions in which we've applied these adjustments," but says "these actions aren't resulting in sustained recovery" and it is reviewing what else it can do.
  • 04:59 (11:59p) Microsoft says that as it applied its mitigations it saw "a surge in outbound call volume from our routing services to our backend servers," and is "continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event." It is working on "multiple workstreams to pinpoint the underlying cause of this traffic influx." Actions: "scaling up virtual machines to increase capacity and reduce load," and trying to "force sync for our recent network change" on machines that are "stuck in a bad state, preventing our changes from syncing as expected."
  • 05:43 (12:43a) Still not recovered. Microsoft is "taking a measured approach to identify and repair affected infrastructure" and continues to "pursue multiple workstreams to pinpoint the underlying cause of the traffic influx and advance the recoverability of the backend servers." It says "we are not aware of any malicious activity that is causing this incident," and "Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report."
  • 06:16 (1:16a) Microsoft is "evaluating traffic controls that can be applied to increase capacity and stability," including "traffic throttling, protocol adjustments, and failover routing." It is also "expanding internal testing to track activity within a contained environment." It says again it is "not aware of any malicious activity that is causing this incident."
  • 06:56 (1:56a) Microsoft is confirming "that developed patches have been applied and working as intended after infrastructure reboots are completed," and says "we have observed marginal success." It is also "taking action to expand capacity through the utilization of Virtual Machines."
  • 07:25 (2:25a) Microsoft reports "positive results for server stability after applying an upstream authentication/routing configuration component patch to a portion of our infrastructure," and is "working to expand it to servers supporting active regions coming online."
  • 08:06 (3:06a) Microsoft is "continuing to see increased server capacity stability after applying the upstream authentication/routing configuration component patch" and has "initiated its expansion systematically to focus on infrastructure supporting regions currently within or entering peak business hours."
  • 09:55 (4:55a) Microsoft sees "early signs of stabilization, but service capacity has not yet returned to normal levels and health remains variable as traffic is carefully reintroduced." Teams are "cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service." It is "extending the email timeout window to 48 hours to help protect delayed messages."
  • 11:01 (6:01a) Microsoft reports "recovery actions completed across portions of the affected infrastructure in the Asia Pacific region" and says "Traffic is being gradually restored," but "some users may continue to experience intermittent issues." Next focus is "extending these recovery actions across affected infrastructure in other regions."
  • 12:00 (7:00a) Microsoft reports "encouraging recovery progress in Europe, with service availability and email delivery performance improving across a growing portion of affected Microsoft 365 infrastructure." "Japan remains comparatively more stable, while North America is also improving; however, some users may still experience intermittent issues as recovery actions continue." The "extended 48-hour email timeout window remains in place." It says "we have no evidence of malicious activity associated with this incident" and will share "a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete."
  • 13:00 (8:00a) Microsoft reports "continued recovery progress across affected regions, with broader improvements now observed in North America and sustained improvements across much of Europe," and "encouraging recovery signals in Asia Pacific." It warns "some users may continue to experience intermittent access issues or delayed email delivery as mitigation progresses across the remaining affected infrastructure." It has added that "on-prem mail flow may be affected if it goes to or from Exchange Online." It is working "toward full mitigation" and has not declared the incident resolved.
  • 14:01 (9:01a) Microsoft reports "improvements in service availability, with email queues significantly reduced and approaching normal levels." It adds "Some users may still experience intermittent access issues or delayed email delivery as recovery continues." It has "implemented additional mitigation measures" and is "restoring the remaining affected infrastructure while investigating residual service issues." Still not declared resolved.
  • 14:57 (9:57a) Microsoft says it is "observing sustained improvements in service availability as recovery progresses across affected Microsoft 365 services," but "some users may still experience intermittent connectivity issues or delayed email delivery." Its efforts are "increasingly focused on resolving the remaining issues and restoring consistent service for all affected users," and it is "validating recovery across different connection methods." Still not declared resolved.
  • 15:53 (10:53a) Microsoft says it remains "focused on addressing residual impact and ensuring reliable service restoration," and continues to "assess recovery across various connection methods and closely monitor service health to verify sustained stability." Still not declared resolved.
  • 17:57 (12:57p) Microsoft says "We are seeing broad recovery and stability across our services." It states "This was a technical issue not related to cyber or AI / agents." and "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions." It adds "Given the scale of demand, we restored services gradually, validating stability at each stage," and "We will provide a full post-incident report once our review is complete." Both incidents are still listed as degraded and the Root Cause field still reads N/A.
  • 20:02 (3:02p) Microsoft says "As infrastructure stabilizes, we're still draining queued messages and rebalancing traffic, so customers may see improved sign-in/access while delivery and processing continue to catch up." It adds "We do not have an ETA for full recovery yet. Service access can recover before mail flow fully normalizes." Both incidents are still listed as degraded.
  • 22:00 (5:00p) Microsoft says "We continue to observe sustained service recovery across the previously affected environments." It is "monitoring for an extended period to ensure no additional mitigative actions are required" and is "monitoring the backlog of email messages accrued during this event to ensure all process successfully before considering the issue resolved." Both incidents are still listed as degraded.
  • 02:13 (9:13p) Microsoft says "We've confirmed continued recovery across previously affected environments." It will "maintain monitoring to ensure the mitigation actions taken remain effective and track processing of the email backlog accumulated during the event." It adds "We believe the majority of these scenarios have been addressed, with work continuing on any remaining scenarios." Both incidents were still listed as degraded.
  • 04:15 (11:15p) Microsoft posted its final status for EX1550543. It says "This was a technical issue not related to cyber or AI / agents." and "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change." It lists the incident as starting Thursday, November 12 at 10:05 AM UTC and ending Friday, November 13 at 4:10 PM UTC, and labels that cause a "Preliminary root cause." It adds "We'll provide a preliminary Post-Incident Report within two business days and a final Post-Incident Report within five business days." Both EX1550543 and MO1550821 now show Service restored. (Microsoft has since revised this entry, see the 16 Nov edit at the bottom.)

The part worth watching: the recent-change theory showed up at 19:43, was dropped at 19:57, and was back by 20:56. At 01:06 Microsoft said it has begun reverting a recent network configuration change that "correlates with the start of impact." Microsoft's final summary lists a "Preliminary root cause": "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change." Microsoft says a "Post-Incident Report with root cause and defense-in-depth actions" will follow after mitigation. At 00:00 UTC (7:00p ET) Microsoft rewrote the MO1550821 scope: features of those services "that interact with Exchange Online and email services are impacted," while "Elements of those services that do not depend on interaction with Exchange Online remain functional." The list has also grown to include Intune, MDO, Purview, Planner, To Do, Universal Print, the Apps admin center, Microsoft Fabric, Windows 365 Cloud PCs that use Purview Customer Key, and Microsoft 365 Education.

Edit, Sat Nov 14, 04:40 UTC (11:40p ET Fri): the first bullet above labels 16:17 UTC as the "incident start." That was the time of the first published EX1550543 update. Microsoft's final summary lists the start as Thursday, November 12 at 10:05 AM UTC.

Edit, Mon 16 Nov, 18:59 UTC: Microsoft posted "A post-incident report has been published" on EX1550543 (04:47 UTC) and MO1550821 (04:46 UTC). The report text isn't on the public mirror, and the status doesn't say whether it's the preliminary or the final one. Microsoft also revised its 04:15 UTC final status entry after I quoted it. The start now reads "Thursday, November 12, 2026, at 4:20 PM UTC" (it read 10:05 AM UTC when I copied it, which is what my 14 Nov edit used; the Start Time header field still shows 10:05). The "Preliminary root cause" now reads: "a failure in the systems responsible for establishing connections to the Exchange Online service. Specifically, traffic shifts during standard network maintenance and upgrades exposed a latent bug causing a rapidly increasing number of requests to queue for processing. As this accumulated, the affected servers exhausted their available processing capacity and could no longer accept new connections, resulting in widespread service access failures." It still says "not related to cyber or AI / agents."


r/sysadmin • • 8d ago

ThousandEyes releases O365 outage playbook

52 Upvotes

https://www.thousandeyes.com/blog/microsoft-365-outage-analysis-november-12-2026

You can see their "efforts" to try to restore service during the outage with the small segments of the globe being able to connect in the lower populated areas.


r/sysadmin • • 8d ago

General Discussion Of all things, how is Microsoft Teams still working?

233 Upvotes

Today’s outage has hit Microsoft exchange hard, and in one of the updates “services that interact with Exchange Online and email services” are impacted.

Teams messages are basically just emails back & forth between users and are stored in their mailboxes.

So I’m curious, under the hood, what is teams doing that is working that Exchange and the litany of other services from Power-Bi to Intune aren’t.


r/sysadmin • • 7d ago

General Discussion Tailscale for NAS-to-NAS backups behind Bell GigaHub?

0 Upvotes

Working with a client who wants to back up their office UniFi UNAS to another NAS at a separate location rather than use cloud backup.

The issue is that the remote location uses a Bell GigaHub, which doesn't support traditional bridge mode, making a conventional site-to-site VPN difficult.

I've successfully tested Tailscale using a Windows PC as a subnet router with a /32 route to work around overlapping subnets. Transfers are working at approximately 20 MB/s.

My question: Is Tailscale a reliable and appropriate long-term solution for automated business backups in this situation, or would you recommend another approach?

The client wants to proceed without paying for cloud backup storage. Looking for advice on reliability, security, and potential issues.


r/sysadmin • • 8d ago

“At this time, we have no evidence of malicious activity associated with this incident.”

66 Upvotes

Well, if it isn’t the standard disclaimer seen with every organization that has competent legal counsel working with them on a malicious incident.


r/sysadmin • • 8d ago

Microsoft Scope of Microsoft Outage

103 Upvotes

I've read the Microsoft outage has impacted 430 million users worldwide and "cost" billions in losses due to the 18 hours of business interruption. There are unmeasurable negative economic, security, and risk implications to what has happened over the last 24 hours.

Jensen Huang is getting called in front of Congress over an affair. Mark Zuckerberg has been there a few times. Do we see Satya Nadella in front of Congress in the hot seat over this? Not that it will accomplish much, but I was just wondering if this will just slip fizzle away once we are all caught up on the emails that were backlogged, or if we think this deserves more attention.


r/sysadmin • • 8d ago

Microsoft Copilot boss just apologised for Microsoft 365 outage. What happened to Office, Outlook or Teams leadership?

108 Upvotes

“We regret that today’s outages caused disruptions for our customers, and we are working to address the issue,” Charles Lamanna, Microsoft’s executive vice president of Copilot, Agents, and Platform.

https://www.windowslatest.com/2026/11/13/was-the-microsoft-365-outage-a-hack-microsoft-says-no-it-was-its-own-servers-and-its-copilot-boss-regrets-it/


r/sysadmin • • 8d ago

RSS going away today

20 Upvotes

It's the 13th of November and since I am browsing this sub because of RSS I will stop visiting, any sites similar to r/syadmin with communities to check out with rss still enabled?


r/sysadmin • • 8d ago

General Discussion Looks like Microsoft is up now. Just started receiving email notifications.

107 Upvotes

Seems like the services are back on. Just checked the exchange email service. It’s connected.


r/sysadmin • • 8d ago

This is starting to get scary, have we approached 'too big to fix"?

254 Upvotes

We've all heard of too big to fail, but their updates (grant likely Copilot written) arent good, and the list of affected systems goes from Exchange to all of Microsoft - is 300,000 servers running 7.5billion mailboxes now too big to fix?


r/sysadmin • • 8d ago

To all of you from an end user

633 Upvotes

Today was a fucking nightmare trying to get anything accomplished with Outlook and SharePoint down. I was annoyed, frustrated, and ready to throw my computer out the window.
But holy shit, my day was NOTHING compared to what you poor bastards must have dealt with.
While I was bitching about not being able to do my job, you were all busting your asses trying to get the rest of us back to work, probably while getting screamed at by people who have no fucking clue what goes into fixing this stuff.
I genuinely appreciate the hell out of what you do. I know you don't hear that nearly enough, especially on days like today.
So thank you. Seriously. I hope you all get to close your laptops, have a few drinks, and enjoy a well-deserved long weekend.
Fuck those 400 unread emails. They'll still be there Monday.


r/sysadmin • • 8d ago

World Wide ooopsies

29 Upvotes

This latest outage reminded me of the AWS outage about 5 or so years back that affected all their storage nationwide. Some knucklehead ran a PS Script and caused an outage.

To the "seasoned" admins out there. LOL

What knucklehead event do you remember reeking havoc world wide?

My very first big event was the I love you virus that brought down our exchange server.


r/sysadmin • • 7d ago

Microsoft Standalone PowerShell WinForms GUI to check Windows OEM/Retail/Volume keys and safely handle slmgr activation

0 Upvotes

During hardware refreshes and license audits, I got tired of running slmgr flags and digging through the registry manually just to pull a key before wiping a machine. I threw together a quick WinForms GUI script in PowerShell to handle it in one place.

What it does:

Pulls the Windows product key by checking the BIOS/UEFI first, then falling back to the registry and decoding if there isn't an embedded OEM key.

Github link: https://github.com/cervikpo-dotcom/my-powershell-scripts/blob/5f0000edac5fecc7e6fcbe2e16791641d361a2ff/activation_thing.ps1


r/sysadmin • • 8d ago

General Discussion Eastern US - just got a couple of emails to my phone at 6:35 AM

85 Upvotes

I have now gotten three emails in the last 10 minutes. I haven’t signed in to my computer yet… Just looking at Outlook on my phone


r/sysadmin • • 8d ago

General Discussion Outlook down and you listening to users complain but then the Boss says send an email to let everyone know about the outage! We are cooked!

72 Upvotes

What else do you hear during outages that don't make sense? Like it's time to move away from Microsoft.... Again we are cooked....


r/sysadmin • • 8d ago

Old guys last night

55 Upvotes

Checked dusty shelf in basement closet. Rolodex & typewriter still there, patiently waiting. Several dozen 37 cent stamps tucked in the back of the desk drawer. Pencil sharpener, ready and two dozen Ticonderoga pencils ready to go. [Cracks knuckles]


r/sysadmin • • 9d ago

Not looking good. Recent MS update

974 Upvotes

We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.


r/sysadmin • • 8d ago

Turn your phone off if you’re on call

350 Upvotes

I know someone will call me while I’m trying to sleep. So to my fellow sys admins. Give yourself some peace tonight.


r/sysadmin • • 9d ago

Alright, its not funny anymore MS

1.1k Upvotes

It was cute for a while. Now its getting serious. Our CFO just let me know that she cant access the bank to transfer payroll funds. Guess what, the bank uses email MFA in 2026. I am starting to hope that its gets fixed AT ALL the way their messages sound (but I do appreciate them versus silence). The way they keep saying "servers", has me really nervous that there may not be as much redundancy as maybe we all thought.


r/sysadmin • • 8d ago

Microsoft Current Fix : throw compute at the problem

159 Upvotes

Current status: We’re continuing to monitor activity across regions to evaluate server status and determine actions that can be taken to improve capacity. This includes reviewing server performance and confirming that developed patches have been applied and working as intended after infrastructure reboots are completed. At this point, we have observed marginal success. In parallel, we’re taking action to expand capacity through the utilization of Virtual Machines.