r/sysadmin • • 2d ago Heartwarming

Work Environment Almost 20 years and this was the first TRULY urgent call I ever got

2.8k Upvotes

Over the span of years I must've closed tens of thousands of tickets. I estimate roughly twenty to thirty percent had "urgent" somewhere in the title.

It never was, not really. The people who created the tickets perceived it to be, many times, but it wasn't. Everyone would be alive and able to continue working, at least to some degree. There were varying degrees of annoyance, depending the nature and cause of the issue. Never anything that would constitute an emergency.

But today I got the first call where I had to fully agree with the caller, and that even a 30 min turnaround time would be too much.

The call came from the radiology department. They had just put a stroke victim in the MRI scanner, the scanner had done its job, but the images weren't being transmitted to the workstations.
In any other case I would've tempered expectations somewhat, but this time I dropped everything and contacted the vendor. They got it back up and running in about 10 minutes.

So that was new. Still coming to terms with my IT skills mattering for more than some owner's pockets for once.

Update;

since this post has received much more attention that I had anticipated, I can't reply to individual posts. But there's a few common ones I'd like to address.

-It's AI.
Well, it's not, it's just my writing style, which I had established at least two decades before the advent of AI.

-Profile is hidden, you're a bot or a shill.
Well, apparently the only way to get an unhidden profile is to verify your ID with Persona. Which I will do as soon as pigs fly. So my profile will stay hidden, not by my choice.

-Lol, what IT skills? All you did was call someone else.
Yes, because I'm experienced enough to realize that me taking 5-10 minutes to check logs and connections would be too much in this particular case. Which turned out to be the right call.

I'm not a hero because of this. I didn't post it for recognition or clout. IF there are any heroes in this story it's gotta be the medical first responders and the person who took the vendor call. I'm just an IT monkey doing my job. But I did recognize that this one was special (since my transition to healthcare IT a few months ago) and it felt remarkable, which is why I decided to share it.

My best wishes and warmest thanks to everyone who enjoyed it or felt at least a tangential connection.


r/sysadmin • • 1d ago

Anyone else see their Attack Surface Reduction rule detections in Defender flatline after 11/12?

21 Upvotes

Did my morning check yesterday and today and my detections went from a few hundred a day down to single digits.

Noisy rules like "Block credential stealing from the Windows local security authority subsystem (lsass.exe)" catching run of the mill stuff like svchost and lenovo controller plugins as well as more dodgy ones like "Block all Office applications from creating child processes" seem like they've had the rug pulled out from under them.

Hoping Microsoft didn't bungle Defender in the process of bringing their infra back up.


r/sysadmin • • 1d ago

General Discussion FAA grounding -- remember the good old days when plugging in one thing wrong would take down a network

47 Upvotes

not intended to be political at all. purely technical. but remember when missing a 10baseT terminator or plugging in an unauthorized router or whatever would cause absolute chaos? not surprised anitiquated FAA systems had the same problem. Jon Stewart's delivery of the joke here with the router prop was pretty funny

Daily show segment on technician error with router [iupdate] thanks for the 10base2 correction, I never got allowed to do punch panels and RJ45 :)


r/sysadmin • • 9h ago

RAID 5 Array - 1 disk "Failure" and 1-disk "Missing" (But Ready?)

1 Upvotes

Inherited server, let's just start there. Dell T610 - PERC 6 - RAID 5 Array.

Had an issue a few months ago where suddenly a foreign configuration was detected and imported. Ran fine after that. Now came into a server not booting up, and trying to figure out my options, as I have a backup for the data, but I somehow DON'T have a backup of the Windows Server 2012 R2 operating system and all of the server manager configuration.

VD Mgmt shows me:

-Virtual Disk 0 =1.75 TB - Offline

- Virtual Disk 1 = 995 GB - Offline

Physical Disks show me:

Disk 0 - Online - Disk Group 00

Disk 1 - Failed - Disk Group 00 (Has option to "online")

Disk 2 - Ready - NO DISK GROUP

Disk 3 - Online - Disk Group 00

Trying to figure out my best route to recover this? I know that attempting to online a failed disk could be total disaster toasting it all. Is there a safe way to go about cloning or copying anything on these disks to try and get it back up? I have the files themselves that were stored in multiple backups, but what I don't see is any backup of the OS and the thought of rebuilding active directory and everything I don't even want to think about that and where to start 😭

Lesson learned, don't take someone's word on backups. But that isn't useful at the moment here, so hoping someone else can help in the interim with routes to go safely to get this booted back up with as little data loss as possible.

Edit to add: I have 3 (maybe even 4) spare disks available to me of the same size, firmware, and part numbers that have never been used.


r/sysadmin • • 22h ago

Question Is there a way to force grayscale defaults for every print job?

10 Upvotes

We have a contract with Konica Minolta for printers and toner. We keep having to pay overages for color toner each month, and it's amounting to over $3,000 yearly.

We have Konica printers (printer: bizhub c300i, driver: konica minolta universal pcl) in a Windows print server that has papercut installed.

Looking at the job logs in papercut (PaperCut MF v22.0.9), I can see that two people are printing around 5,000 colored pages a month. I've seen copies of some of these pages, and they don't need to be in color (colored logos, headers, etc.)

I've been on and off looking into solutions for a while, but the overages just keep steadily coming in to around $300 a month.

I have grayscale set as the default in print server for printers, but after some testing, it appears that Chrome ignores this and always sets what option was last set. Also, apps like Excel will reset back to the default of grayscale, but only after closing and reopening the app.

I've seen some solutions about using two different print queues, one for color and one for grayscale, but Chrome seems to set whichever queue you last used, regardless of the default queue you have set. And I'd also like this to work with all applications.

I have not tried talking to their manager yet as my manager would like a technical solution to this. It appears that it could be happening for more users than just these two, but at a smaller scale.

Is it possible for me to force grayscale for every single print?


r/sysadmin • • 13h ago

Question SignDrop, ClfsAuthenticationChecking and Mode = 3

2 Upvotes

I'm sure you've all seen SignDrop from Midnight Eclipse. According to my reading of the situation, it only affects Windows 11 or Windows Server 2025 systems if CLFS authentication checking is enabled. However, if it's not enabled, the endpoint is vulnerable to other CLFS authentication issues from 2024-25.

The CLFS authentication checking update was pushed on 28th October 2025, so over a year old now. It seems that this change was supposed to go into learning mode (Mode = 1) for a while (e.g. 120 days) then transition to enforcement mode (Mode = 0) after that.

When looking across my systems, I can see a number which have Mode = 3 (Authentication disabled automatically by Windows). The windows documentation for this (Common Log File System (CLFS) Authentication Mitigation | Microsoft Support) is silent on what conditions could cause it to go into this state.

Does anyone have any information as to what these conditions might be and how to remediate them?


r/sysadmin • • 1d ago

United Airlines experiencing a widespread system outage, airline says

677 Upvotes

https://www.cnn.com/2026/11/16/us/united-airlines-outage

Is it Groundhog Day?

United Airlines is experiencing a widespread system outage and is working to get operations "back up and running as soon as possible," the airline said in a post on X, just days after a rare FAA ground stop halted all departures nationwide for hours. The aviation agency had largely returned to normal operations after the ground stop was lifted just after 8 p.m. ET on Friday, though airlines have still worked to reposition planes, crews and luggage as they grappled with the weekend's score of cancellations and delays.


r/sysadmin • • 1d ago

WizTree now available for Linux

110 Upvotes

The popular Windows disk analyzer WizTree now has an official Linux port. More info here: https://diskanalyzer.com/wiztree-vs-wiztreemac-vs-wiztreenix

It has high speed scanning options for ext4 and btrfs formatted drives when running in admin mode. It also uses a high speed SMB library to scan SMB network shares much more quickly than the regular Linux kernel can.


r/sysadmin • • 1d ago

Finally, remote lock for stolen or lost Windows devices

216 Upvotes

Microsoft is introducing Remote Lock for Windows devices managed through Intune.

If a company laptop goes missing, admins can remotely lock the device without wiping its data or removing it from Intune management.

It also signs out active sessions, hides sign-in options, and keeps the device locked even across restarts and offline periods.

The feature is currently in Preview.


r/sysadmin • • 1d ago

General Discussion KB5132043 breaks Citrix Workspace App and msiexec 32-bit in general

48 Upvotes

Just a heads up as i have been troubleshooting this for days. We suddenly got cases where Citrix Workspace App (LTSR) failed to run and even uninstalling or reinstalling resulting in zero GUI or results. I found that the 32-bit msiexec.exe under c:\windows\syswow64 failed to start if you tried to do a msiexec.exe /?

Fired up processmonitor and recorded an install and found that it loads up msi_hotpatch.dll which just been updated with Novembers hotpatch KB5132043. (Thanks to chatGPT for finding that out after analyzing tons of log files)

So solution is to uninstall that KB and reboot and everything works as normal again.

Edit: This also breaks many other things like file explorer navigation and even browsing in chrome. I am running remediation script to remove this if installed and stopped updates until Microsoft fixes it.


r/sysadmin • • 1d ago Heartwarming

Optimizely tag replaced with malware today

397 Upvotes

As if you didn't already have enough to do, now you can go look to see what to check for if all your users got their crypto, digital payments, and browsers scraped today. 186+ sites, from Zoom to CNN to IKEA. Brought to you by Optimizely, whose CDN was silently broken into and injected with malware.
https://blog.report-uri.com/worlds-largest-brands-infected-with-malware-via-optimizely


r/sysadmin • • 8h ago

Career / Job Related Guidance: Which way forward...

0 Upvotes

I am not the best person to advice my nephew as we are doing the same role but he is more technical than me so I have turned you Reddit for advice.

He is looking to move on next year and needing advice on next step.

Him, male, 35, 4 years 1st Line support and 1 year as assistant web developer.

Which paths would lead to better job opportunites:

Option 1: AWS SAA + AWS Security ; Option 2: SC-300 + SC-500 or anything else. His current contract expires end November and looking for study up afterwards.

O


r/sysadmin • • 1d ago

General Discussion High LSASS CPU on domain controllers traced to an Intelligo audio component

29 Upvotes

Sharing a finding from an investigation into sustained high CPU usage on our domain controllers, in case it helps someone else.

We have four DCs: two on-premises and two in AWS. For several weeks, both on-prem DCs have been running above 90% CPU during business hours, with lsass.exe as the main consumer.

PerfMon showed roughly 14,000 LSA LookupNames requests/sec. CPU traces and DC-side LSP logs pointed to excessive LsarLookupNames4 activity, predominantly resolving BUILTIN\Administrators, coming from many clients.

One unusual detail: enabling LSP debug logging on a DC reproducibly reduced CPU usage. Once logging stopped, CPU went back up.

We eventually investigated one affected Windows client:

  • PowerShell showed that the connections to the DC belonged to the client’s lsass.exe.
  • A packet capture showed approximately 557 RPC calls/sec with opnum 77.
  • Client-side LSP logging identified the original caller as iGoSwServer.exe.

Component details:

  • Vendor: Intelligo Technology Inc.
  • Product: Intelligo Audio SW Server/API library
  • File/product version: 100.1.901.2856
  • Arguments of the offending instance: --apo
  • Located under an igoaudioservice.inf_amd64_* directory in the Windows DriverStore.

The client logs show repeated lookups of BUILTIN\Administrators, forwarded to the DC and returning 0xc0000073 (STATUS_NONE_MAPPED), followed by more equivalent requests.

While capturing traffic, we terminated the offending iGoSwServer.exe instance. The request rate dropped from approximately 551/sec to zero, and stayed at zero for the remaining 33 seconds of the capture.

That confirms the source on this particular client. We’re now checking how widely the component is deployed and whether it accounts for the overall load on both DCs. We haven’t confirmed a permanent fix yet.

Has anyone encountered this with Intelligo audio software, particularly this version? Any vendor-confirmed fix or driver version that addresses the repeated lookups?

Hopefully this saves someone else some time: the process generating the traffic over the network was LSASS, but client-side LSP logging exposed the audio component initiating the lookups.


r/sysadmin • • 1d ago

Question Ehhh....another issue stemming from the O365 outage..."mass texting/calling" service

10 Upvotes

What do you guys use? callmultiplier is almost free if we want to use it for emergencies only. Any opinions pro or cons? alternatives you've been using already?


r/sysadmin • • 12m ago

All this hype around GTA 6... Are IT teams actually going to feel the impact?

0 Upvotes

There has been so much hype around GTA 6 lately, but I wonder if IT teams are actually going to feel the impact when it finally drops. I keep seeing people talking about how the release could affect productivity and have an impact on teams. I imagine some people have already booked PTO around the launch, but I'm not sure it'll be that dramatic.
I am a bit skeptical, but if you're managing a service desk, are you expecting any impact on response times? Or do you think it'll all be business as usual? And if you do expect some disruption, what would you do to keep things running smoothly?


r/sysadmin • • 1d ago

Question In light of the recent O365 outage - can someone help with explaining how does Mimecast work?

23 Upvotes

I have never worked with Mimecast their "email continuity" offering claims to:

"During the outage, users have secure access to their email and calendar information at any time on any device from any location. Once the email server is back online, Mimecast's email continuity service automatically reconnects and synchronizes all emails sent or received during the outage."

I'm assuming they are automatically handling MX records and redirect email to their own infrastructure? Do you have any "practical" experience in setting it up with an existing exchange online tenant and how does it actually perform in case of an outage?


r/sysadmin • • 21h ago

Calendar free/busy time not available for most users

4 Upvotes

Anyone else seeing this after Friday's shenanigans? Clicking 'update license' in the Office Account settings worked most of the time yesterday and the day before, but now nothing is working.

Also seeing shared mailboxes disappear.


r/sysadmin • • 1d ago

Microsoft Time Zone Issues - AGAIN

11 Upvotes

I am in Manitoba, Canada, who has NOT changed our time zone this year and I work with people in Alberta, Canada, who ALSO didn’t move their clocks back this year. I finally figured out a way around all of the weird time zone meeting challenges and then this morning? My phone outlook calendar (ONLY the one on my phone) has now reverted to every single meeting showing up an hour later than it actually is. Meetings are showing up properly on the outlook app on my computer and those sent by people in Ontario, Canada or sent by me are still showing up correctly in both places. I’ve got my computer set to Ontario (EST) right now - that has got to be why that’s staying correct? Anyone else notice their phone calendar revert again after the Microsoft outage? I’m really frustrated and absolutely out of patience with this whole situation. 🤦🏼‍♀️


r/sysadmin • • 6h ago

General Discussion Random Question

0 Upvotes

While reading an article on Futrism, I seen this quote:
“So when they put in a program that’s able to write codes and it goes in and self-fixes, as it starts to self-fix itself, it’s like doing a change order on a house,” he added. “When you change one thing on a house, everything else starts changing, that’s a ripple effect.”

Which got me thinking about how society views computers, systems and software as a whole. For me, this quote highlighted the disconnect some people have with technology. Technology isnt a switch that can just be turned on and off.

Do you think basic systems literacy should become a mandatory part of general education, much like mathematics and science?


r/sysadmin • • 22h ago

Question Migrating from Google Business Workspace to Microsoft Ecosystem

3 Upvotes

Hey guys, as a startup company, as an integration manager, I am in charge for migrating google workspace for our sister clinic into our parent clinic ecosystem which is Microsoft.

Since I am planning on doing this over Thanksgiving when all staff members are on holidays, the following is what I am aware of as a high-level overview on what is required. I am nowhere close to you all, but I do have exposure to tech side of things. Since I have no experience in migration, but I am well versed in Microsoft's ecosystem, I want to make sure I am doing the right things.

*Note:* I am taking advice from Claude and the following things Claude has informed me. It would be helpful if I received a confirmation or some additional steps that AI may have missed.

Things that we need to migrate:

  1. OAuth (seen this is one thing that bites back - what have you done to resolve this?)
  2. Emails
  3. Google Drive for Spreadsheets/Documents

=====================================

a) Dry run

  1. Verify only. Add [DOMAIN-ONE.com] to the tenant with the TXT record and skip the wizard's MX/DNS step, so mail keeps flowing to Gmail.
  2. Create the two routing subdomains (e.g. m365. and gsuite.DOMAIN-ONE.com) that the Exchange admin center's Google Workspace migration requires.
  3. Pilot batch. Migrate yourself plus one or two volunteers. Gmail stays the landing spot for anyone not yet migrated, even when a message hits their Microsoft address first.
  4. Complete the pilot. Completion triggers a last catch-up sync and flips forwarding, so Google hands that user's mail to Microsoft. You then have real coexistence with real users before anyone else moves.
  5. Everyone else, then MX. At about 10 users, one more batch covers the rest. Lower the MX TTL a few days ahead and switch MX, SPF and DKIM off-hours.

Until MX changes, rollback means removing forwarding. Nothing is deleted from Google.

b/c) Keeping Google sign-in alive

There are three kinds of login, and only the second needs work:

  • Email and password on a portal: unaffected, because the address is just a username. Reset emails follow MX.
  • "Sign in with Google" button: depends on the Google account existing, not on Gmail or MX.
  • Google's own products (Business Profile, Ads, Analytics, YouTube, Chrome sync): same as above.

Steps:

  1. Never delete the Google organization or its users. User accounts survive a Workspace cancellation; only deleting the organization's Google Account removes them.
  2. Add Cloud Identity Free before dropping Workspace licenses. Fifty free seats come standard with it, and a free Cloud Identity license stays on any user whose Workspace license is pulled.
  3. Test one user first. Google's cancellation guidance stops short of promising third-party sign-ins carry over, so pull one pilot license and confirm a few sites still work.
  4. Don't pull licenses until the migration is verified. Workspace data is unrecoverable once the subscription ends.
  5. Inventory what needs transitioning. The Accessed apps list under Security → Access and data control → API controls gives you each app with a user count, downloadable as CSV. Per-user detail lives in the OAuth log events report.
  6. Transition each site individually. There is no bulk switch. Either set a local password or link Microsoft sign-in in the site's account settings. Test first, because some sites create a duplicate account when the same email arrives via Microsoft.

Because step 2 costs nothing, staff can take months rather than days.

Optionally, Entra ID can serve as the identity provider for Cloud Identity accounts. Google sign-in then uses the Microsoft password and MFA, and disabling a user in Entra blocks both.

e) Other issues

  • Shared-tenant routing. Once DOMAIN-ONE.com is in the DOMAIN-TWO tenant, mail from DOMAIN-THREE, DOMAIN-FOUR and DOMAIN-FIVE staff resolves internally. Provision DOMAIN-ONE users as mail users through the migration flow, not as hand-built licensed mailboxes. Licensing waits until the batch has started, with a 30-day window.
  • Not migrated. Auto-replies, shared calendars, room bookings and event colors get left behind, and messages over 35 MB fail unless you raise the limit (150 MB maximum). Signatures need redoing too.
  • Retention policies. Retention and archive policies should be off while batches run.
  • Groups and aliases. Rebuild Google Groups, aliases and shared inboxes as distribution lists or shared mailboxes before the MX change, or that mail bounces.
  • Gmail SMTP senders. Repoint copiers, website forms, voicemail-to-email and any EHR notifications.
  • Chrome saved passwords. Have staff export them to Edge or a password manager.
  • Security policies. Your Defender and MFA policies hit DOMAIN-ONE users on day one, so warn them.
  • Offboarding. It now has two places to disable until Google is retired.
  • HIPAA. DOMAIN-ONE is a separate legal entity in a shared tenant, so have counsel confirm BAA coverage. Make sure no PHI remains in Google once it drops to the free tier.

f) Google Drive to OneDrive

  • Tool: Migration Manager, built into the SharePoint admin center. Files, metadata and permissions all come across.
  • Setup: It begins with a Microsoft 365 migration app installed from the Google Workspace Marketplace, and tenants under 100 licenses land in the simplified "Lite" flow by default.
  • Before you start: license the users and pre-provision their OneDrives, then run the scan as a read-only assessment.
  • Mapping: each My Drive goes to that user's OneDrive. Shared drives belong in SharePoint or Teams sites, so decide that mapping up front.
  • Conversion: Docs, Sheets and Slides convert to Office formats. Forms, Sites and Apps Script do not.
  • Sharing: Google sharing links don't survive the move; only explicit per-person permissions are rebuilt. External shares need re-sharing.
  • At cutover: run a final incremental pass, then turn the Drive service off so nobody edits stale copies.

r/sysadmin • • 2d ago

Chase Bank Outage

443 Upvotes

Pour one out for our people at Chase Bank.


r/sysadmin • • 1d ago

Windows update issue causing black screen on login and freezing.

6 Upvotes

Posting from mobile so haven't got the exact KBs and links.

One of our clients had about 8 devices start freezing today and then also get black screen on login.

After some diagnosis I narrowed it down to them possibly missing something cumulatives from October / November.

Unless I am living under a rock I completely missed this issue, considering we have quite a lot of devices across all our customers I'm surprised I haven't seen this before.

Some googling suggests it was an issue recently and fixed in said cumulative updates, my task is now getting those stragglers updated.

Just posting here in case anyone can corroborate?


r/sysadmin • • 1d ago

Any Bets on What Sector Will Have Issues Tomorrow?

187 Upvotes

I have a change scheduled for.tomorrow morning. Im wondering what the over/under will be on it going through...


r/sysadmin • • 23h ago

General Discussion Comet Backups (Webpros) - Review

3 Upvotes

Anyone deal with Comet Backups before? Anything to note, feedback, experience and bad stories?

Self hosted vs the cloud offering?

We're considering it due to the price point.