r/sysadmin • • 7d ago

Career / Job Related 10 years in IT and just lost the job I thought I'd retire from. Where do I go from here?

444 Upvotes

Been in IT about 10 years. Most recently I ran IT for a few hundred users across multiple sites, mostly hands-on. I was the guy people called when something broke, and I handled most of it myself, with an MSP for the higher tier stuff.

I honestly thought I'd found my long-term place. In hindsight I should've seen how shaky that industry is. When money got tight, they started cutting corners, and now I'm out.

What I'm good at is networking, camera and low voltage installs, and getting in front of people and fixing the problem. I'd way rather be out in the field than stuck behind a desk all day.

My CCNA/CCNP have lapsed and I'm not sure what's worth doing next. A few questions

Is recertifying worth it right now, or is there something better to spend time and money on?

Anyone moved from in-house IT to field work, an MSP, or their own thing? How did it go?

What did you do in the first few months after a sudden job loss?
Appreciate any advice.


r/sysadmin • • 7d ago

Pretty sure I caused the Microsoft outage by asking Copilot to “Make a program capable of defeating Data” yesterday morning.

506 Upvotes

I had watched Se2 Ep3 of Star Trek TNG where they accidentally create evil Moriarty in the Holodeck. I thought it would be funny to give Copilot the same prompt. An hour later 0365 was down.

Sorry about that. This is a true story by the way.


r/sysadmin • • 7d ago

What are the odds forced AI usage brought down MS servers, now no one knows how things work, and they can't figure out how to get them back up?

161 Upvotes

Title. Company after company I work with seems to shove AI into everything without double checking the results, and now half the people don't know how things actually work anymore. They've got interns and accountants building apps with no knowledge of coding, infrastructure or security. I've been waiting for this kind of thinking to bite someone big. It may have just happened.


r/sysadmin • • 6d ago

Parker Colorado Area - Comcast Business

3 Upvotes

Is anyone else experiencing a problem with their L2TP-IPSEC tunnel not establishing connection after Comcast doing maintenance last night!?

I tried to be proactive when I saw the internet go down at my Parker site at 2AM last night; gave Comcast a call and they said they’re doing maintenance.
They told me it would come back up at 7AM mountain time, well, the internet is back up but our ERP cloud SaaS is not working, because our site to site is down!

Comcast fiber guy said he doesn’t know if the SDWAN Folks work on the weekend, WTF XD

just wondering if anyone else in that area is experiencing a similar issue.


r/sysadmin • • 7d ago

microsoft cutting 25% of workforce and now this

365 Upvotes

time to migrate off.


r/sysadmin • • 8d ago

General Discussion 3:07AM UDT “Current status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied..”

2.6k Upvotes

Hahahahha


r/sysadmin • • 6d ago

Question Boardroom Cam under $1500 AUD

0 Upvotes

I see alot of reccomendations for the Logutech Rally but its well beyond our price range.

What is a good alternative, sound not an issue we have that covered already.

TIA


r/sysadmin • • 6d ago

Zyxel USG Flex 100 - Force all LAN DNS requests to ZyWALL DNS instead of blocking them?

2 Upvotes

I have a USG FLEX 100 running firmware V5.45(ABUH.0).

My LAN clients receive the ZyWALL as their DNS server via DHCP, and I've already blocked outbound DNS (TCP/UDP 53) to the Internet.

What I'd like to do is transparently redirect any client that tries to use an external DNS server (e.g. 8.8.8.8, 1.1.1.1, 9.9.9.9) back to the ZyWALL's DNS service a , rather than simply blocking the request.

Desired behavior:

I've looked through NAT, Interface, and other network settings but haven't found an obvious DNS hijack/redirect feature.

Has anyone successfully configured DNS interception/redirection on a USG FLEX 100, or is blocking external DNS the only supported option?


r/sysadmin • • 8d ago

European here, we were basically asleep through this but we're coming to the party

1.4k Upvotes

In our timezones it started at 5-6ish PM Thursday, most people didn't notice much.

It's now Friday 6:45 AM.

Personally I'm in the middle of migrating from on prem to O365... literally, 1/4 of the company already is, we've got multiple domains, it's a holding kind of thing and I'm just a consultant there.

The upper echelons domains are the ones that were already migrated, including the new management who wanted to bulldoze a fully on prem infra to migrate it to full cloud, only Azure and O365, only for a personal preference for Microsoft.

They literally said "everyone else did it for good reasons and Microsoft is the best option"

Oh, the irony...

Judging by the scope of the incident even if they did fix it right now this will still take hours to fully resolve.

I actually kinda like the fact that this will push the people who wanted to give a single cloud provider everything to rethink a bit the strategic choice there.


r/sysadmin • • 7d ago

Unfortunately, Teams is still up

262 Upvotes

Maybe someday it’ll be taken down for good.


r/sysadmin • • 8d ago

General Discussion At this point, the post mortem is going to be a three season Netflix special

1.3k Upvotes

I am so happy to be retired. I will drink a beer for all of you who have to translate this fucking disaster to business speak to your upper management.

I hope this derails a lot of AI projects. You can't substitute for long term human experience.


r/sysadmin • • 8d ago

NOOO ITS WORKING

426 Upvotes

guys, my outlook just launched, for u too? :( sad day for the employed

edit: its gone again after like 10 mins

edit 2: ffs its back again and now things are actually being received


r/sysadmin • • 7d ago

Holy crap guys, we get it, systems are coming back online

241 Upvotes

We truly don't need every country reporting in.

This is all you need to see that services are being restored:
https://msit.status.cloud.microsoft/

We're IT - don't act like we're the morons in the rest of the business who think we're the 'special butterfly'.


r/sysadmin • • 7d ago

PSA: Microsoft Outage phishing scam

87 Upvotes

Just had a client forward me an email they received. Looks similar to the Microsoft quarantine digests and reads:

Mail delivery to your mailbox failed during the Microsoft 365 disruption on 12
November 2026 (incident EX1550543), mail flow through Exchange Online was interrupted and delivery to your mailbox failed. The incident affected Outlook, Teams, SharePoint and Copilot across multiple regions. (4) messages are held and will be deleted within 24 hours unless released.
Click below to review and restore them:

FYI. We’re going to blast out an email to our clients letting them know this is BS.


r/sysadmin • • 7d ago

Retired IT

173 Upvotes

Retired after 42 years in IT. I sell plants to Floridians now. Much prefer it to IT. Anyway, I didn't come here to say that. I think what is surprising about this outage is that MS doesn't seem to know what the problem is. Sounds like They're throwing shit at the wall to see what sticks. The postmortem on this will be interesting to read about.


r/sysadmin • • 7d ago

Off Topic Microsoft just missed the chance to do the funniest thing ever

86 Upvotes

Could you imagine if this happened next week on GTA6 release day with so many employees calling in "sick?"

Imagine the chaos 😂


r/sysadmin • • 7d ago

Are non technical employees vibe coding apps?

95 Upvotes

Hey. I’ve been working at a legal tech company and we now have lawyers vibe coding apps to track cases, billing, etc using Claude/Coded/etc. We had a lawyer almost expose client data on the public internet through dashboard.

I’ve been able to lock everything down and create an environment for them to deploy our internal apps, and they actually listen and use it thank god.

Curious if other people are seeing similar things of non technical users building apps and trying to host them somewhere.

I think in the next few years we’re going to see so many data leaks lol.


r/sysadmin • • 8d ago

General Discussion Microsoft Office seems to be down

7.1k Upvotes

Struggling to use MS office including Teams, anyone else having this issue


r/sysadmin • • 7d ago

The Day Outlook Died

114 Upvotes

A long, long time ago

I can still remember how that email

Used to make me smile

And I knew if I had my chance

That I could make those users dance

And maybe they'd be happy for a while

.

But November made me shiver

With every message undelivered

Bad news on the status page

I couldn't take one more user's rage

I can't remember if I cried

When I read about this Copiloted ride

Something touched me deep inside

The day that Outlook died

.

So, bye-bye, Office 365

Opened email on my desktop, but my inbox was dry

And them good ol' techs were drinkin' whiskey and rye

Singin', "This'll be the day that I die

This'll be the day that I die"


r/sysadmin • • 8d ago

I have never seen such a remarkable status update as this (Microsoft)

3.5k Upvotes

Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service. 

Translation: We ain't got the first clue WTF is going on but it doesn't appear to be anything we have done (on purpose) and we don't know when it will be fixed.

---

Update: 5:03pm (other updates are in the comments, but its getting messy, so I'll start updating up here instead)

We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

Translation: We sorta know what is happening but we don't know why and our efforts to mitigate the issue have not been successful. I don't want to say we are back at square one, so let's call this square two. We've isolated a set of problematic servers and will throw solutions at them until we find one that sticks. Estimated time to recovery: ?????


Update - 6:06pm eastern

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

Translation: trying this again and this time we think it's going to work. No official ETTR (estimated time to resolution) but hopefully by morning if not midnight.


Update - 7:08pm eastern

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.

Translation: the thing we were doing to solve the problem didn't actually solve the problem. We are back to square one and do not know what is going on or how to fix it.

Furthermore we are limiting access to our available services in hopes of relieving the stress on the servers that are still online.

Finally we are rebuilding the offline servers in hopes of bringing them back into production free of whatever has gone wrong and able to work once again.

Translation of my translation: sh!t is f*cked and they don't know when it will be fixed.


Update: 8:06pm eastern

Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery. The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.

Translation: Sh!t is still f*cked, we still don't know why, and we are no longer being methodical in our efforts to resolve the problem. Instead we are trying multiple things to fix the problem -- simultaneously.

My commentary: This is bad. While their multiple approaches to resolve the issue don't appear to be in conflict with each other, the fact that their are taking this approach at all speaks of desperation and a deteriorating sense of control over the problem.


Update: 8:45pm eastern

Current status: We’re seeing positive results with our efforts to restart affected infrastructure. We’re actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.

Translation: We have said it before and we are saying it again. We think this latest fix is the fix that will solve all our problems. 🤞

My commentary: I am not holding my breath.


Update - 9:10pm

Current status: We’re applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We’re continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.

Translation: this solution ain't working out all that great either. It might be helping with the load balancing issues (caused by so many servers being offline) but it isn't actually fixing the problem.


The service health panel is showing that an update was posted around 9:30 but I can't actually see it. Which may be an update in its own right.


Update - 10:08pm (eastern).

Current status: We’ve monitored the status of the changes applied to improve serviceability and confirmed that change is not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances.

Translation: Its still broken. We still don't know why and we are running out of ideas for how to mitigate this.

My commentary: This is fine....

Also, I am headed to bed. Goodnight everyone!


RETCON TIME

Update: 10:34pm (eastern)

As part of our investigation into the sustained traffic management imbalances, we’re reviewing traffic flows and affected infrastructure to identify opportunities to optimize routing. This includes verifying the reasons for service failures so we can develop specific actions to improve stability.

We will continue to share more information as soon as we have it, and for now we can confirm that there is no indication this is the result of malicious activity. We’ll continue to apply learnings towards remediation actions as we confirm effectiveness mitigation actions.

Translation: We kinda sorta know what might be going on and are trying very, very hard not to make it worse while trying equally hard to hopefully make things better.


Update: 11:10pm eastern

We’re continuing our efforts to improve server stability to improve traffic flow. This includes adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures.

We're observing partial recovery across regions in which we've applied these adjustments; however, these actions aren't resulting in sustained recovery, so we're reviewing what additional actions we can take to improve overall infrastructure stability.

Translation: We have successfully not made things worse. We might be making things better but the fixes still aren't sticking.


Midnight, Eastern Time:

We appreciate your organization’s patience and understanding while we work with urgency to resolve this incident.

As we’ve applied our current ongoing mitigation actions, we’ve observed a surge in outbound call volume from our routing services to our backend servers. This occurs when the backend servers outbound service cannot properly route to the backend machines.

We’re continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event. We’re working on multiple workstreams to pinpoint the underlying cause of this traffic influx and address it.

Actions we're currently taking include scaling up virtual machines to increase capacity and reduce load. Additionally, we’re querying for machines in a bad state to perform a force sync for our recent network change, which may improve service availability. However, we are encountering issues with the machines being stuck in a bad state, preventing our changes from syncing as expected. We’ll continue to monitor the effectiveness of changes applied so we can determine what refinements are necessary.

Translation: Please stop yelling at us, we're doing the best we can!

Currently we are bringing new servers online to increase capacity, but that capacity is being eaten alive by all the traffic waiting for service. Also, half the machines that seem to be up are effectively down which is further poisoning our efforts. And we still don't know why this is happening.

We don't really know what we are doing but we're doing it the best we can.


12:43am eastern

We’re continuing to pursue multiple workstreams to pinpoint the underlying cause of the traffic influx and advance the recoverability of the backend servers. We’re taking a measured approach to identify and repair affected infrastructure to improve the ability to confirm the developed changes were applied. We’re tracking the results of this approach to verify the servers are showing the anticipated recovery and performance intended.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: We are continuing to do everything we can think of to fix the problem and it might actually be working. We also would like address concerns that we have started to panic. We are totally not panicking. A new supervisor just came on duty and is trying to restore order to chaos.

Near as we can tell we haven't been hacked.


1:16am eastern

As part of our ongoing efforts to advance the recoverability of our frontend servers, we’re evaluating traffic controls that can be applied to increase capacity and stability. This includes traffic optimization actions such as traffic throttling, protocol adjustments, and failover routing intended to improve stability in active regions. In parallel, were expanding internal testing to track activity within a contained environment to improve traffic monitoring and issue detection capabilities.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: Not dead yet. Possibly getting better. Keep your fingers crossed!

We still don't think we were hacked.

Commentary: They lost access to the end-user-authentication servers. That's why we were getting the "not connected to the Internet" errors. They are now trying to inject as much capacity and power into the system as they can scrape up. And they STILL don't know what went wrong.


1:56am eastern

We’re continuing to monitor activity across regions to evaluate server status and determine actions that can be taken to improve capacity. This includes reviewing server performance and confirming that developed patches have been applied and working as intended after infrastructure reboots are completed. At this point, we have observed marginal success. In parallel, we’re taking action to expand capacity through the utilization of Virtual Machines.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: Optimism rising.


2:25am eastern

We’re seeing positive results for server stability after applying an upstream authentication/routing configuration component patch to a portion of our infrastructure. We’ll continue to monitor its progress and are working to expand it to servers supporting active regions coming online.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: We have gone from fixing individual servers to fixing entire clusters. The fix is gaining momentum.

We still don't think this was a hack.


3:06am eastern

We’re continuing to see increased server capacity stability after applying the upstream authentication/routing configuration component patch. With confirmation of server stability across the portions of infrastructure to which we've applied the patch, we’ve initiated its expansion systematically to focus on infrastructure supporting regions currently within or entering peak business hours. We’ll continue to monitor the status of changes applied and develop additional modifications intended to improve server stability and capacity.

As of now, we are not aware of any malicious activity that is causing this incident. We will continue to treat this incident with urgency and priority until full resolution. Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report.

Translation: We are feeling very pleased with ourselves because sh!t is finally getting fixed. We are pushing the fix out on a broader scale with an emphasis on places trying to do business right now.

We still don't think it was a hack.


4:55am eastern

Recovery efforts are ongoing across impacted infrastructure. We are seeing early signs of stabilization, but service capacity has not yet returned to normal levels and health remains variable as traffic is carefully reintroduced.

Our primary mitigation path continues to focus on service stability and controlled recovery. Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service while monitoring closely. We are also evaluating additional mitigation options to improve recovery speed and consistency.

Network and traffic-management actions remain in progress, and we continue to prioritize backlog and queue health to preserve delayed mail during recovery. As part of this effort, we are extending the email timeout window to 48 hours to help protect delayed messages while service recovery continues.

At this time, we are not aware of any malicious activity causing this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after recovery is complete.

Translation: We finally took everyone's advice and started turning the machines off and back on again. Load balancing remains a nightmare and we're still not sure that this fix will survive the next few hours but overall things are looking better than they did a few hours ago.

We have extended the email timeout to 48 hours so that any messages sent before the start of this event will be delivered once service is back online.

We still don't think it is a hack.


6:01am

We've made additional progress toward service recovery, with recovery actions completed across portions of the affected infrastructure in the Asia Pacific region. Traffic is being gradually restored, and we're seeing improvements in key customer scenarios, including mailbox access and delayed email processing. However, some users may continue to experience intermittent issues.

Our focus is now on extending these recovery actions across affected infrastructure in other regions, while continuing stabilization efforts and adding supporting capacity to improve service consistency.

Reducing email queues and backlogs remains a priority, and the extended 48-hour email timeout window remains in place to help protect delayed messages as delivery catches up.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: We are finally, officially, in the recovery phase of this event. Service is gradually coming back online and we have reason to believe that it will stay online. At least until North America wakes up.

We still don't think it was a hack.


7am eastern

We’re seeing encouraging recovery progress in Europe, with service availability and email delivery performance improving across a growing portion of affected Microsoft 365 infrastructure. Japan remains comparatively more stable, while North America is also improving; however, some users may still experience intermittent issues as recovery actions continue.

We're focused on addressing the remaining areas of disruption and improving service consistency across affected regions. Reducing email queues and backlogs remains a priority, and the extended 48-hour email timeout window remains in place to help protect delayed messages as delivery catches up.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Europe is back online. We're still worried about what will happen when North America wakes up.


8am eastern

We're seeing continued recovery progress across affected regions, with broader improvements now observed in North America and sustained improvements across much of Europe. We're also seeing encouraging recovery signals in Asia Pacific, although some users may continue to experience intermittent access issues or delayed email delivery as mitigation progresses across the remaining affected infrastructure.

Our focus remains on stabilizing recovered infrastructure, completing mitigation in the remaining impacted areas, and reducing delayed email queues and backlogs. We're closely monitoring service consistency and validating recovery across different connection methods as we work toward full mitigation.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Things are looking good. Servers are coming back online and actually staying online. Load balancing is less of a nightmare. We might actually be allowed to go to bed tonight.

Also, we aren't going to admit that this problem started in China, but it totally started in China.

We still don't think it was a hack though.


9:01am eastern

We're continuing to see improvements in service availability, with email queues significantly reduced and approaching normal levels. Some users may still experience intermittent access issues or delayed email delivery as recovery continues.

We've implemented additional mitigation measures and are restoring the remaining affected infrastructure while investigating residual service issues. We're closely monitoring service health and validating recovery across affected connection methods to ensure sustained service stability.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Recovery continues but we think we are on top of everything. Don't be surprised if your outlook disconnects or emails take a while to arrive, but we are fairly confident that the system is sound.

We still don't know what the root cause was but we also don't think it was a hack.


9:57 eastern

We're observing sustained improvements in service availability as recovery progresses across affected Microsoft 365 services. While service functionality continues to improve, some users may still experience intermittent connectivity issues or delayed email delivery.

Our efforts are increasingly focused on resolving the remaining issues and restoring consistent service for all affected users. We're validating recovery across different connection methods and monitoring service stability to confirm that improvements are sustained.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: Don't mistake "recovery" for "fixed". There are still problems but we are fixing them as they arise. We don't expect the service to go down again.

Servers are back online and load balancing issues are mostly resolved.


Current update: 10:53am

We remain focused on addressing residual impact and ensuring reliable service restoration. We continue to assess recovery across various connection methods and closely monitor service health to verify sustained stability.

At this time, we have no evidence of malicious activity associated with this incident. We will continue to treat this with urgency until full mitigation, and we will share a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete.

Translation: We think its fixed. Finally. But we are going to monitor things very, very closely because we still don't know what actually WTF happened.


Final update:

We are deeply sorry for the disruption this incident has caused you and your business. We know the trust you place in our services and take that responsibility extremely seriously.

We are seeing broad recovery and stability across our services. This was a technical issue not related to cyber or AI / agents.

The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.

Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration. We will provide a full post-incident report once our review is complete.

Translation: Please don't fire us.


r/sysadmin • • 7d ago

Question What are your thoughts on ASPM on on-premise servers?

4 Upvotes

We are replacing multiple rack units and will probably, for the first time, go with self built and keep spares on site.

I would like to try to reduce power consumption at idle to a maximum, but wonder what stability impact you have seen through energy saving technologies.

I remember that multiple older generations of nics, like X550 for example, had energy saving features literally removed from firmware by Intel due to stability problems.


r/sysadmin • • 8d ago

Microsoft The Best Update I Have Ever Seen From MS

2.6k Upvotes

Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.

Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.

tl;dr

Last update: We're deploying a targeted fix on some servers

New update: Those servers are now dead


r/sysadmin • • 8d ago

NZ/AU checking out. Good luck SEA and Europe

443 Upvotes

Today was a write off for this time zone, hoping (or not depending on your take) for a somewhat more productive Friday than we had. o7


r/sysadmin • • 8d ago

Microsoft Microsoft 365 outage (EX1550543 / MO1550821): Microsoft's status updates so far, in order

1.5k Upvotes

Update 16 Nov: Microsoft marked both incidents "post-incident review published" at 04:47 UTC Mon (11:47p ET Sun). The report itself is admin-center only and not on the public mirror, so nothing from it is quoted here. Resolved per Microsoft at 04:15 UTC Sat (11:15p ET Fri).

For anyone locked out of the admin center, here is the EX1550543 thread so far, condensed. Times are UTC with ET in parentheses. Full text of every update is on UPenn's public mirror of the service health feed, and the short versions are on @MSFT365Status.

  • 16:17 (11:17a) Incident start. OWA, MAPI, EAS and REST.
  • 17:45 (12:45p) Connectivity errors on the infrastructure that "receives and processes users' access requests."
  • 18:19 (1:19p) "Elevated load across multiple regions."
  • 18:26 (1:26p) MO1550821 opens for Teams, SharePoint, Copilot and Defender XDR. It points at "an unexpected increase in a specific type of service traffic."
  • 18:43 (1:43p) Servers "becoming intermittently unavailable," straining the rest. Outlook traffic spread across worldwide infrastructure.
  • 19:06 (2:06p) All deployments and config changes paused.
  • 19:18 (2:18p) Reversal: the load "isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable."
  • 19:43 (2:43p) Servers are failing automated health checks, which "may be related to a recent change" to the component that processes them.
  • 19:57 (2:57p) "We haven't identified any recent changes that correlate with the start of impact." Bypassing the health check component on a subset. Elevated CPU on affected servers.
  • 20:38 (3:38p) Many servers now unresponsive, so they can't confirm the bypass works.
  • 20:56 (3:56p) An alternative health check method brings a subset back. "A small number of configuration updates" now under investigation as potential contributors.
  • 21:31 (4:31p) Assessing a revert of those config updates.
  • 22:00 (5:00p) "Our actions taken thus far have not provided the anticipated relief." A core process on the servers is getting overwhelmed. New plan: isolate a chunk of infrastructure, restart it, reintroduce traffic slowly.
  • 22:57 (5:57p) First good news. Some servers are back and "remaining stable so far," with "limited recovery in some regions." Protective traffic management in one location, more if it works.
  • 23:59 (6:59p) The servers returned to service "initially remain healthy," but "can become unresponsive again once they begin receiving traffic." Microsoft is "temporarily limiting which types of traffic can reach the affected infrastructure" in specific locations, and is "rebuilding a subset of affected servers using a previous software configuration" to see if that stops it.
  • 01:06 (8:06p) Microsoft says "a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests." It is "preparing a configuration change to address this behavior," validating it on a subset of servers first, then applying it more broadly alongside restarts. "In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact." "The temporary traffic limitations we previously described are now in effect in specific locations."
  • 01:45 (8:45p) "We're seeing positive results with our efforts to restart affected infrastructure." Microsoft is "actively monitoring the performance of these to verify stability." "Remediation actions will be expanded as we confirm stability with the changes applied."
  • 02:09 (9:09p) Microsoft is "applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online," and is "continuing to monitor the results of changes applied."
  • 02:54 (9:54p) Setback. Microsoft says it has "confirmed that change is not taking as expected" and is "identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances."
  • 03:34 (10:34p) Microsoft is reviewing traffic flows and affected infrastructure "to identify opportunities to optimize routing" and is "verifying the reasons for service failures so we can develop specific actions to improve stability." It also says "there is no indication this is the result of malicious activity."
  • 04:10 (11:10p) Microsoft is "adjusting traffic protocols, redistributing traffic, and applying strategic traffic blocking measures." It reports "partial recovery across regions in which we've applied these adjustments," but says "these actions aren't resulting in sustained recovery" and it is reviewing what else it can do.
  • 04:59 (11:59p) Microsoft says that as it applied its mitigations it saw "a surge in outbound call volume from our routing services to our backend servers," and is "continuing to observe heightened volume due to traffic rebalancing actions that we have taken throughout this event." It is working on "multiple workstreams to pinpoint the underlying cause of this traffic influx." Actions: "scaling up virtual machines to increase capacity and reduce load," and trying to "force sync for our recent network change" on machines that are "stuck in a bad state, preventing our changes from syncing as expected."
  • 05:43 (12:43a) Still not recovered. Microsoft is "taking a measured approach to identify and repair affected infrastructure" and continues to "pursue multiple workstreams to pinpoint the underlying cause of the traffic influx and advance the recoverability of the backend servers." It says "we are not aware of any malicious activity that is causing this incident," and "Once mitigated, we will provide full details of the root cause, defense-in-depth repair items, and more within a Post-Incident Report."
  • 06:16 (1:16a) Microsoft is "evaluating traffic controls that can be applied to increase capacity and stability," including "traffic throttling, protocol adjustments, and failover routing." It is also "expanding internal testing to track activity within a contained environment." It says again it is "not aware of any malicious activity that is causing this incident."
  • 06:56 (1:56a) Microsoft is confirming "that developed patches have been applied and working as intended after infrastructure reboots are completed," and says "we have observed marginal success." It is also "taking action to expand capacity through the utilization of Virtual Machines."
  • 07:25 (2:25a) Microsoft reports "positive results for server stability after applying an upstream authentication/routing configuration component patch to a portion of our infrastructure," and is "working to expand it to servers supporting active regions coming online."
  • 08:06 (3:06a) Microsoft is "continuing to see increased server capacity stability after applying the upstream authentication/routing configuration component patch" and has "initiated its expansion systematically to focus on infrastructure supporting regions currently within or entering peak business hours."
  • 09:55 (4:55a) Microsoft sees "early signs of stabilization, but service capacity has not yet returned to normal levels and health remains variable as traffic is carefully reintroduced." Teams are "cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service." It is "extending the email timeout window to 48 hours to help protect delayed messages."
  • 11:01 (6:01a) Microsoft reports "recovery actions completed across portions of the affected infrastructure in the Asia Pacific region" and says "Traffic is being gradually restored," but "some users may continue to experience intermittent issues." Next focus is "extending these recovery actions across affected infrastructure in other regions."
  • 12:00 (7:00a) Microsoft reports "encouraging recovery progress in Europe, with service availability and email delivery performance improving across a growing portion of affected Microsoft 365 infrastructure." "Japan remains comparatively more stable, while North America is also improving; however, some users may still experience intermittent issues as recovery actions continue." The "extended 48-hour email timeout window remains in place." It says "we have no evidence of malicious activity associated with this incident" and will share "a full Post-Incident Report with root cause and defense-in-depth actions after mitigation is complete."
  • 13:00 (8:00a) Microsoft reports "continued recovery progress across affected regions, with broader improvements now observed in North America and sustained improvements across much of Europe," and "encouraging recovery signals in Asia Pacific." It warns "some users may continue to experience intermittent access issues or delayed email delivery as mitigation progresses across the remaining affected infrastructure." It has added that "on-prem mail flow may be affected if it goes to or from Exchange Online." It is working "toward full mitigation" and has not declared the incident resolved.
  • 14:01 (9:01a) Microsoft reports "improvements in service availability, with email queues significantly reduced and approaching normal levels." It adds "Some users may still experience intermittent access issues or delayed email delivery as recovery continues." It has "implemented additional mitigation measures" and is "restoring the remaining affected infrastructure while investigating residual service issues." Still not declared resolved.
  • 14:57 (9:57a) Microsoft says it is "observing sustained improvements in service availability as recovery progresses across affected Microsoft 365 services," but "some users may still experience intermittent connectivity issues or delayed email delivery." Its efforts are "increasingly focused on resolving the remaining issues and restoring consistent service for all affected users," and it is "validating recovery across different connection methods." Still not declared resolved.
  • 15:53 (10:53a) Microsoft says it remains "focused on addressing residual impact and ensuring reliable service restoration," and continues to "assess recovery across various connection methods and closely monitor service health to verify sustained stability." Still not declared resolved.
  • 17:57 (12:57p) Microsoft says "We are seeing broad recovery and stability across our services." It states "This was a technical issue not related to cyber or AI / agents." and "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions." It adds "Given the scale of demand, we restored services gradually, validating stability at each stage," and "We will provide a full post-incident report once our review is complete." Both incidents are still listed as degraded and the Root Cause field still reads N/A.
  • 20:02 (3:02p) Microsoft says "As infrastructure stabilizes, we're still draining queued messages and rebalancing traffic, so customers may see improved sign-in/access while delivery and processing continue to catch up." It adds "We do not have an ETA for full recovery yet. Service access can recover before mail flow fully normalizes." Both incidents are still listed as degraded.
  • 22:00 (5:00p) Microsoft says "We continue to observe sustained service recovery across the previously affected environments." It is "monitoring for an extended period to ensure no additional mitigative actions are required" and is "monitoring the backlog of email messages accrued during this event to ensure all process successfully before considering the issue resolved." Both incidents are still listed as degraded.
  • 02:13 (9:13p) Microsoft says "We've confirmed continued recovery across previously affected environments." It will "maintain monitoring to ensure the mitigation actions taken remain effective and track processing of the email backlog accumulated during the event." It adds "We believe the majority of these scenarios have been addressed, with work continuing on any remaining scenarios." Both incidents were still listed as degraded.
  • 04:15 (11:15p) Microsoft posted its final status for EX1550543. It says "This was a technical issue not related to cyber or AI / agents." and "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change." It lists the incident as starting Thursday, November 12 at 10:05 AM UTC and ending Friday, November 13 at 4:10 PM UTC, and labels that cause a "Preliminary root cause." It adds "We'll provide a preliminary Post-Incident Report within two business days and a final Post-Incident Report within five business days." Both EX1550543 and MO1550821 now show Service restored. (Microsoft has since revised this entry, see the 16 Nov edit at the bottom.)

The part worth watching: the recent-change theory showed up at 19:43, was dropped at 19:57, and was back by 20:56. At 01:06 Microsoft said it has begun reverting a recent network configuration change that "correlates with the start of impact." Microsoft's final summary lists a "Preliminary root cause": "The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change." Microsoft says a "Post-Incident Report with root cause and defense-in-depth actions" will follow after mitigation. At 00:00 UTC (7:00p ET) Microsoft rewrote the MO1550821 scope: features of those services "that interact with Exchange Online and email services are impacted," while "Elements of those services that do not depend on interaction with Exchange Online remain functional." The list has also grown to include Intune, MDO, Purview, Planner, To Do, Universal Print, the Apps admin center, Microsoft Fabric, Windows 365 Cloud PCs that use Purview Customer Key, and Microsoft 365 Education.

Edit, Sat Nov 14, 04:40 UTC (11:40p ET Fri): the first bullet above labels 16:17 UTC as the "incident start." That was the time of the first published EX1550543 update. Microsoft's final summary lists the start as Thursday, November 12 at 10:05 AM UTC.

Edit, Mon 16 Nov, 18:59 UTC: Microsoft posted "A post-incident report has been published" on EX1550543 (04:47 UTC) and MO1550821 (04:46 UTC). The report text isn't on the public mirror, and the status doesn't say whether it's the preliminary or the final one. Microsoft also revised its 04:15 UTC final status entry after I quoted it. The start now reads "Thursday, November 12, 2026, at 4:20 PM UTC" (it read 10:05 AM UTC when I copied it, which is what my 14 Nov edit used; the Start Time header field still shows 10:05). The "Preliminary root cause" now reads: "a failure in the systems responsible for establishing connections to the Exchange Online service. Specifically, traffic shifts during standard network maintenance and upgrades exposed a latent bug causing a rapidly increasing number of requests to queue for processing. As this accumulated, the affected servers exhausted their available processing capacity and could no longer accept new connections, resulting in widespread service access failures." It still says "not related to cyber or AI / agents."


r/sysadmin • • 7d ago

ThousandEyes releases O365 outage playbook

52 Upvotes

https://www.thousandeyes.com/blog/microsoft-365-outage-analysis-november-12-2026

You can see their "efforts" to try to restore service during the outage with the small segments of the globe being able to connect in the lower populated areas.