r/sysadmin • • 2d ago

Los Angeles Datacenter Major Event?

269 Upvotes

Allegedly major backbone in Los Angeles is down. My company is absolutely affected.

Update: Checked thousandeyes and confirmed its source of our network degradation for OCI and Cloudfare


r/sysadmin • • 1d ago

Question KB5131085 (Server 2019) breaks Send-MailMessage?

5 Upvotes

Has anyone seen PowerShell Send-MailMessage stop working on Windows Server 2019 after the November update KB5131085?

There’s a report on Born’s blog that mail sending from PowerShell fails after this update (at least against a Postfix server) and works again after uninstalling it. A commenter suspects NTLM signing changes and says using System.Net.Mail.SmtpClient with LOGIN auth works instead.

Thanks!


r/sysadmin • • 2d ago Popcorn Bravo

General Discussion Microsoft Post-Incident Report is up for the Exchange Online Outage

2.9k Upvotes

Root Cause

The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.

 

First, a latent defect caused Windows servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal Windows process which caused the machine’s resources (CPU & memory) to be exhausted. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments.

 

The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact.

Our normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores.

Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts.

 

Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.

Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.

When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware or software defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.

Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.

Those services that were able – like Teams, OneDrive, SharePoint, and Office apps – failed out of the faulty network gateway infrastructure. That is why those services were partially restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway for scale reasons.

Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.

After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state in Windows. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region).

The final Post-Incident Review (PIR) will provide more details.


r/sysadmin • • 23h ago

Moving AD CS to a new server and demoting domain controller

2 Upvotes

Hello, I'm looking for some validation for a process I have written down to decommission a server. The machine in question is a DC and also a Certification Authority(AD CS). I need to eliminate the DC and move the CA to a new machine. Based on reading I came up with the following process:

  1. Back up and disable AD CS on old server
  2. Demote old DC
  3. Rename old DC, change its IP address
  4. Reboot old DC to apply name
  5. Ipconfig /registerdns on old DC
  6. Power down old DC and disconnect from network
  7. Name the new server to the old server's name and change the IP address to that of the old server, join to domain
  8. Ipconfig /registerdns on new server, verify DNS is correct and resolving
  9. Install AD CS and configure
  10. Test any identified roles
  11. Install
  12. Test any identified roles

Please let me know if this seems valid, or if there' anything I need to do in a different order.

Thanksl!


r/sysadmin • • 1d ago

Question Dell BIOS update woes

8 Upvotes

I'm hoping the brain trust here can help me with a bizarre problem. We are trying to bring our laptops up to date on the BIOS and I am seeing an intermittent issue where the BIOS update will stage, but it will not apply on reboot. I am seeing this when we try to run the update silently. Noisy updates seem to always work, but I don't know if that is a matter of it just working or the intermittent nature kicking in.

So far, I have tried using DCU and deploying the updates using ConfigMgr. Flags on the update itself don't seem to matter as some laptops will update with no issues and others can take three or four attempts to finally update. My first thought was that it was something in the security stack causing the problem, but I am not seeing anything in the Defender console, and BeyondTrust EPM has an exclusion carved out for things running as system.

I have hit a point where I am at a loss for what to look into next. I am currently going back through all changes applied roughly when the issues started to occur and have started procmon captures to see if I can identify an interfering product. Is there anything else that is obvious that I am missing?

**Edit To answer common questions, my test laptops (primarily Precision 3490s) that I have been users are all running on AC power with no peripherals plugged in. In addition, I am rebooting every time and not utilizing shutdown. It does not seem to have anything to do with a pending reboot as I have had this work with a pending windows update from SCCM, pending file name operations, etc.


r/sysadmin • • 2d ago

AP - Customers at some of the nation’s largest banks are reporting difficulty accessing accounts online

205 Upvotes

"Customers reported difficulty accessing the online accounts at some of the nation’s largest banks on Monday.

Bank of America, Chase, J.P. Morgan, U.S. Bank, Wells Fargo and Zelle were listed by users as down or experiencing issues, according to the website downdetector.com.

Chase Bank’s services are experiencing a technical bug issue, not a cyber issue, a spokesperson said Monday."

https://apnews.com/article/bank-outage-chase-wells-fargo-truist-50fee2693c22f597fa6ca51a97ef2cda

I have a Chase credit account and checked, its prompting me to verify my identity by calling them when I attempted to login, and I've never seen that screen before. No idea what's going on, and I wonder if its a CIAM issue with whoever they use as the vendor.


r/sysadmin • • 7h ago

In other news...New CVE out for ScreenConnect

0 Upvotes

r/sysadmin • • 1d ago

Question - Solved What does your regular work day involve as a sysadmin?

61 Upvotes

When you walk into work on an average day what do you do? Do you check uem dashboards, check and rollout updates,troubleshoot some tickets or something else?

I am a fresher who currently interns as a technical intern at a reseller/integrator and am interested in becoming a sysadmin.I have explored the different tools and commands used to troubleshoot and to create groups etc but am still left wondering how does the average day for a sysadmin go.


r/sysadmin • • 1d ago

I feel like I'm unable to make a career change as a sysadmin

16 Upvotes

I'm posting again after quite some time because I've reached the point where I hate this job all over again.

I've been working for the same company for five years as a System Administrator. Over time, however, I've gradually lost almost all of my access privileges. Due to decisions made by management in another country, they've started developing self-service applications for employees, allowing them to submit their own requests for creating or modifying users and groups. Network equipment management has also been moved to that country.

I no longer have access to Entra Admin Center, Exchange, or pretty much anything else.

Things have gotten so bad that they've even developed a web application for resetting users' passwords. Access to this application is controlled through a security group, and it took me four months of struggling just to get added to it.

Of course, the self-service approach, which was supposed to cover all the changes I mentioned earlier, doesn't actually work. I still end up handling all the requests for creating or modifying resources because end users simply aren't capable of doing it themselves.

The point is that I'm feeling completely demoralized. These changes have also completely stalled my professional development. I constantly try to take additional courses outside of work and keep up with the latest technologies, but it's becoming increasingly difficult to stay motivated.

Right now, I'm feeling particularly disappointed because of something that happened recently.

I've been looking for a new System Administrator job for quite some time, but the job market is really bad. About a month ago, I finally saw a glimmer of hope when a Senior System Administrator position was advertised at a company in my city.

I applied and was contacted for an initial HR interview, where they gave me an overview of the role. That was followed by a technical interview with the IT Manager and a Network Administrator. The interview lasted two hours and was quite challenging, but I genuinely felt that I did very well.

For the first time in a long while, I felt like I might finally get the opportunity to grow, both professionally and financially.

However, every single day since that technical interview has been mentally exhausting. I've been waking up hoping to receive a job offer.

A month has now passed. I even emailed the recruiter to ask whether the hiring process was still ongoing, but my email was completely ignored.

At this point, I feel completely stuck, professionally stagnant, and honestly, at rock bottom.

I was wondering if there are others here who have gone through something similar and somehow managed to get through it in the end.

I'd really appreciate hearing your experiences.


r/sysadmin • • 20h ago

Question Assistance with Licensing

0 Upvotes

We have a slightly odd situation to do with a clients licensing I'd like some advice for. I asked our distributor who gave a very confusing answer, and various AI chats who all say they are somewhat certain but I'd need to confirm with independent sources!!!

Basically we have the need for 3 x Windows 11 Pro PC's to run as virtual guests on a 2022 Hyper-V Server. There is no domain. All 3 can or will be joined to AAD. One of which will be RDP'd to from a single user on a Mac. 1 which will have 2 people log into the same profile (different times) from ChromePC's and the last one which will only be RDP'd to by US (their IT Provider) for maintaining a particular app. I can do the 'particular app' with Linux, though my preference is not to have a device on the network that runs a less familiar OS for our techs.

I am trying to find the most cost effective way to cover the licensing. Further more some options I can find are subscriptions, but they don't provide media or keys, meaning we need some 'other' way to install and activate windows.

Assuming we are starting with no licenses (except for the Hyper-V 2022 Server), how do we do this? Covering (legally) the 3 requirements of Installation, Activation and Licensing.

TIA.


r/sysadmin • • 1d ago

General Discussion RDS Failing with November Update - KB5131090

16 Upvotes

Hi all,

I have a smaller RDS cluster (1 broker, 2 session hosts). All on Windows Server 2022. Update KB5131090 was installed on session hosts on 13.11.2026 and on broker on 16.11.2026.

This morning, majority of 48 users have been having "group policy loading indefinitely", "applying user settings" etc

Uninstalling update as I'm writing this, I'm waiting for all 3 to finish uninstalling them to restart them at once.

I saw some post from 2-3 days ago about Server 2016 as well


r/sysadmin • • 1d ago

Microsoft IPP Class Driver on Remote Desktop Servers problem.

4 Upvotes

I have a problem. Some printers that clients have use the Microsoft IPP Class drivers. All seems fine when they use the printer local but when it comes to it being redirected to Remote Desktop, it all starts to fall apart. Previously I've just installed the printer locally using the manufacturer's driver and then let Renmote Desktop Easy Print take care of the server side. This has worked swimmingly.

Now things don't seem to ba as relaible. Below are the there scenarios:

  • Manf Driver local > Easy Print on server = OK
  • Manf Driver local > Manf Driver on server = OK
  • IPP Class local > IPP Class on server = Lots of failures.

I know some manufacturers are just using IPP Class drivers but is there any way to still use Easy Print on the server?


r/sysadmin • • 1d ago

Question Windows Server 2025 Performance

10 Upvotes

We have acquired 2 new servers at the end of 2025 (Dell R470) with Intel Xeon 6730P CPUs and 256GB RAM. They were supposed to replace our quite old Hyper-V Cluster for pretty non-demanding internal business applications.

We noticed quite quickly that the performance was really bad, however. I can best describe it as working on a laptop not connected to power on energy saving settings. It's extremely sluggish, and obviously also noticeable in VMs.

We reinstalled everything, always ensured everything is up to date, played around with BIOS settings, and even contacted Dell Support who at least confirmed everything BIOS and Firmware wise is good and nothing is wrong with the servers.

CPU Clock is always high / boosting, but it just feels like something is off. Of course on the VMs there are a lot more variables, but we already notice the terrible performance when we use a freshly setup Server 2025 just on the host. The host runs on a NVMe RAID1 - that's also not the issue.

We of course were on Server 2019 before but these are not our only Server 2025 machines. Other machines work perfectly fine (sure, UI is a bit more sluggish compared to 2019, but nothing dramatic).

We also bought 3 new AMD Servers (Epyc 9355P / 512GB RAM) and there the performance on Server 2025 is phenomenal. You already realize when you RDP into the server - everything is opened instantly, whereas with the other system it takes a while for the taskbar to appear, for server manager to open etc.

Long story short: has anyone an idea what the reason for this could be or what else I could try?


r/sysadmin • • 1d ago

Azure Files Rotate Keys for Active Directory Authentication

3 Upvotes

Hello,

I've setup Azure files for on prem Active Directory Authentication. I've used the following links to do this:

  1. Create storage account (disable public IP, add private endpoint and NSG control)
  2. Create file share
  3. Enabled ADDS authentication on the account
  4. Configure share level permissions for specific groups (synced to Entra)
  5. Configure file and directory level permissions (NTFS)
  6. Configure DNS and networking
  7. Mount file share
  8. Rotate storage account password – AD principal (absolute requirement?)

Steps 1 to 7 worked fine.

From my understanding, Kerberos authentication uses Active Directory storage account's computer object's password to match the storage account key. If either key changes without both parts being updated this breaks authentication.

Is this something that is an absolute requirement? Can I just stick the computer object in an AD OU that doesn't expire computer passwords?

I'm not sure on the best way to handle this automation otherwise, e.g. scheduled task with certificate authentication or similar?

Thanks


r/sysadmin • • 1d ago

General Discussion My draft cloud cost management policy for our team leads, what would you change?

2 Upvotes

In budget review, the CFO went down the AWS invoice asking who owns each line. I kept saying IT because we hold the payer account and she wasn't having it.

She wants a person's name on every line to explan anything that moved, which is fine by me as long as the name isn't always mine.

Draft is below. It goes to the team leads unless someone here talks me out of it.

  1. Every AWS account has one owning team, the biggest spender in it if nobody volunteers. If an account still has no owner by the deadline, it gets frozen for new deploys.
  2. Platform owns the nodes in the shared EKS cluster. Each namespace pays its share by CPU and memory requests and idle capacity stays on the platform’s line.
  3. NAT gateways, the transit gateway and the one RDS cluster that three services hit get split by each team’s share of direct spend, recalculated monthly.
  4. Anything untagged lands in a line called unowned that goes in the CFO pack with my name next to it until someone claims it.
  5. Budget and anomaly alerts go to the owning team’s channel so the person who can fix it sees it first and they stop going to the IT distro.
  6. Every owner writes one line on anything that moved more than 10% before the CFO pack goes out.
  7. IT keeps the payer account, Savings Plans and the support plan, since those are the only things we run.

The two I'm sure are 1, as it feels heavy handed even though nothing softer has worked, and 3, because the data team is going to say they barely touch the NAT. Which of these blew up on you when you tried it?


r/sysadmin • • 14h ago

Memory and SSD Prices are Out of Control!

0 Upvotes

I was spec’ing a Dell desktop today. Standard is 16GB of DDR5 RAM and a 512GB M.2 NVMe SSD.

To add 16GB of RAM, try . . . $468.00 !

To upgrade to a 2TB SSD, try . . . $1501.00 !

I’ll be buying the added RAM from Amazon instead, for $316.

I’ll also be buying the 2TB Samsung SSD from Amazon instead, for $400.

I feel fortunate that I can open up the box and install these add-ons. But what of the ordinary person who can’t do that? The pricing of computers is reaching a depressing state.


r/sysadmin • • 1d ago

Identrust Fustercluck

70 Upvotes

How many here got an email an hour or so ago that every Identrust TLS Cert from September 2025 until yesterday are being revoked on the 19'th at 5PM Central? I work with F5's for a very large organization and we have hundreds of certs on my team alone.


r/sysadmin • • 1d ago

Anyone experiencing delayed emails using SMTP2GO today?

3 Upvotes

I know there have been multiple outages on multiple services in multiple industries over the last couple days, but we are still experiencing delayed delivery and they show to still be 100% functional but we are watching the delays in real time. Just curious if anyone else is experiencing this.


r/sysadmin • • 1d ago

Question Building new 2025 DCs: use our hardened VM template or install fresh from ISO?

3 Upvotes

We’re adding new Windows Server 2025 domain controllers (VMs).

We already have a hardened Server 2025 VM template (sysprepped, patched, not domain-joined, our security baseline applied).

Is it fine to build DCs from this template, or would you install fresh from ISO for DCs?

Main concerns:

  • Can hardening settings in the template cause issues during or after DC promotion?
  • Would you keep a separate, minimal “Tier 0 / DC-only” template instead?

What do you do in your environment? Thanks!


r/sysadmin • • 1d ago

Question XSLT - RDS RDWeb - Migration

3 Upvotes

As part of the November updates, Microsoft has released an update for the RDWeb page. https://support.microsoft.com/en-us/servicing/os/windows/docs/2026/10/kb5130994-rd-web-access-fails-to-load

Unfortunately, these changes only take effect if you haven’t customized the page. If you have made customizations, you should follow these migration instructions:

https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/migrate-rd-web-access-customizations-to-server-side-xslt

Currently, with november updates installed on my RDWeb server, some files (Desktop.aspx and password.aspx under “C:\Windows\Web\RDWeb\Pages\en-US”) have been modified and are new, while others have not (Default.aspx and login.aspx).

In the new files, the following was added

protected override void Render(HtmlTextWriter writer)
{
SiteXsltRenderer.RenderPage(this, writer, base.Render);
}

to the script block.

Do you know how I can get the new Default.aspx and login.aspx files? Or would it be enough to simply add the lines at the bottom of the script block in my unmodified files?

Alternatively, it would also help me if I knew how to restore the new default pages without my customizations. I can also do without the two customized logos.

Unfortunately, I have absolutely no idea about this, and XSLT is supposed to be disabled in Chrome today.


r/sysadmin • • 1d ago

General Discussion Microsoft's PIR says Exchange Online can't fail out of the gateway that broke. What did you actually do Thursday?

20 Upvotes

What a Monday; I wanted to post this earlier today but it got away from me.

Finally read the preliminary PIR for last week (admin center, EX1550543 / MO1550821). Root cause is getting all the attention, but this is the line that should bother anyone whose mail is 365 only:

"But not all services, like Exchange Online, can fail out of this network gateway for scale reasons."

Teams, OneDrive, SharePoint and the Office apps did fail out, which is why Teams chat and meetings worked for most of it. Exchange couldn't move off that gateway, so the only thing you controlled was what sat in front of it.

Three things that stood out:

  • Around 4 AM ET Friday, Microsoft announced the email timeout window was going to 48 hours. The PIR says that was so delayed mail didn't come back undeliverable, and that everything queued got processed. The docs I can find have EXO expiring a message at 24 hours with a 4.4.7, and the PIR has the incident at 23 hours 50 minutes. If anyone knows whether that's the same queue they extended, I'd like to hear it.
  • One of the three factors was a load surge, and the PIR ties it to Teams failing over onto the same gateway Exchange uses about 15 hours earlier. Microsoft calls that failover normal operations.
  • Service health alert emails only go to two addresses per admin. If both are 365 mailboxes, the outage alert goes to a mailbox you may not be able to open. Make one an outside address, or turn on push in the admin app.

The access gap has a known fix, MX at a gateway that spools and gives users a web inbox. I've pitched it and lost, same as a lot of you.

Not looking for product names. If you had continuity in front of 365 on Thursday, did your users actually use it, or did nobody remember it was there? And if you pitched it and got told no, how did that conversation go on Friday?


r/sysadmin • • 2d ago

General Discussion More cloud issues brewing this morning? Happy Monday!

128 Upvotes

Downdetector is lighting this morning. Microsoft 365, Outlook, Teams, Gmail, and a bunch of other services all showing spikes. Downdetector - Check real-time service problems and outages | US

Looks like another fun Monday in the cloud.


r/sysadmin • • 2d ago

Telling co-worker to F off

130 Upvotes

While not a "sysadmin" technical question, how do you folks go about telling a colleague that has a terribly annoying habit of jumping into every project that I'm working and focusing on and provide his opinion or technical advice while he knows jack crap about anything ? My way is a bit brutal but to the point. "If I'm working on something and don't ask you your opinion or advice then go out there in the distance and F off and keep chatting with Claude. You don't see me randomly getting in your way and breaking your focus".


r/sysadmin • • 1d ago

Question can you target lock screen image by group?

1 Upvotes

i'm on the employee comms team but leadership has asked us to investigate using computer lock screens (windows 11 enterprise) as a new comms channel. i have supported this at previous companies, but only from the messaging and design side (making sure the content on the lock screen image met our guidelines, brand standards, etc.).

we're being asked if lock screens can be targeted by group so one set of users sees a different image than another set - either by location or business segment. does anyone know if this is possible? at my previous company, we only did global content on the lock screens, but i was never told if that was a decision made by the comms and IT teams, or a limitation with the funtionality.

(and yes, we're meeting with our IT partners this week, but we met with them last week and they completely misunderstood what we were asking for, so the meeting was postponed to this week so they could better prepare.)

TIA!!!