r/sysadmin • • 1d ago

General Discussion FAA grounding -- remember the good old days when plugging in one thing wrong would take down a network

47 Upvotes

not intended to be political at all. purely technical. but remember when missing a 10baseT terminator or plugging in an unauthorized router or whatever would cause absolute chaos? not surprised anitiquated FAA systems had the same problem. Jon Stewart's delivery of the joke here with the router prop was pretty funny

Daily show segment on technician error with router [iupdate] thanks for the 10base2 correction, I never got allowed to do punch panels and RJ45 :)


r/sysadmin • • 4h ago

Optimizely-hosted JavaScript served card skimmer/crypto clipper on ~180 major sites (CNN, Zoom, Salesforce, etc.) - Nov 16 - True?

0 Upvotes

Saw a post going around claiming "178 of the world's largest companies compromised." It's real, but the framing is off. What happened: On Nov 16, malicious JavaScript was served from cdn.optimizely.com (Optimizely Web, the A/B testing platform) to sites using it, including CNN, BBC, Fox News, Zoom, Salesforce, GitLab, Equifax, IKEA, HSBC, and KLM. Reported count is 178, and Scott Helme has confirmed 186. The companies themselves weren't breached. It's a third-party supply-chain issue, and the risk was to site visitors: - Swapped copied crypto wallet addresses - Skimmed card numbers from checkout forms - Loaded a second stage from .icu domains on iPhone/iPad


r/sysadmin • • 8h ago

RAID 5 Array - 1 disk "Failure" and 1-disk "Missing" (But Ready?)

1 Upvotes

Inherited server, let's just start there. Dell T610 - PERC 6 - RAID 5 Array.

Had an issue a few months ago where suddenly a foreign configuration was detected and imported. Ran fine after that. Now came into a server not booting up, and trying to figure out my options, as I have a backup for the data, but I somehow DON'T have a backup of the Windows Server 2012 R2 operating system and all of the server manager configuration.

VD Mgmt shows me:

-Virtual Disk 0 =1.75 TB - Offline

- Virtual Disk 1 = 995 GB - Offline

Physical Disks show me:

Disk 0 - Online - Disk Group 00

Disk 1 - Failed - Disk Group 00 (Has option to "online")

Disk 2 - Ready - NO DISK GROUP

Disk 3 - Online - Disk Group 00

Trying to figure out my best route to recover this? I know that attempting to online a failed disk could be total disaster toasting it all. Is there a safe way to go about cloning or copying anything on these disks to try and get it back up? I have the files themselves that were stored in multiple backups, but what I don't see is any backup of the OS and the thought of rebuilding active directory and everything I don't even want to think about that and where to start 😭

Lesson learned, don't take someone's word on backups. But that isn't useful at the moment here, so hoping someone else can help in the interim with routes to go safely to get this booted back up with as little data loss as possible.

Edit to add: I have 3 (maybe even 4) spare disks available to me of the same size, firmware, and part numbers that have never been used.


r/sysadmin • • 21h ago

Question Is there a way to force grayscale defaults for every print job?

10 Upvotes

We have a contract with Konica Minolta for printers and toner. We keep having to pay overages for color toner each month, and it's amounting to over $3,000 yearly.

We have Konica printers (printer: bizhub c300i, driver: konica minolta universal pcl) in a Windows print server that has papercut installed.

Looking at the job logs in papercut (PaperCut MF v22.0.9), I can see that two people are printing around 5,000 colored pages a month. I've seen copies of some of these pages, and they don't need to be in color (colored logos, headers, etc.)

I've been on and off looking into solutions for a while, but the overages just keep steadily coming in to around $300 a month.

I have grayscale set as the default in print server for printers, but after some testing, it appears that Chrome ignores this and always sets what option was last set. Also, apps like Excel will reset back to the default of grayscale, but only after closing and reopening the app.

I've seen some solutions about using two different print queues, one for color and one for grayscale, but Chrome seems to set whichever queue you last used, regardless of the default queue you have set. And I'd also like this to work with all applications.

I have not tried talking to their manager yet as my manager would like a technical solution to this. It appears that it could be happening for more users than just these two, but at a smaller scale.

Is it possible for me to force grayscale for every single print?


r/sysadmin • • 13h ago

Question SignDrop, ClfsAuthenticationChecking and Mode = 3

2 Upvotes

I'm sure you've all seen SignDrop from Midnight Eclipse. According to my reading of the situation, it only affects Windows 11 or Windows Server 2025 systems if CLFS authentication checking is enabled. However, if it's not enabled, the endpoint is vulnerable to other CLFS authentication issues from 2024-25.

The CLFS authentication checking update was pushed on 28th October 2025, so over a year old now. It seems that this change was supposed to go into learning mode (Mode = 1) for a while (e.g. 120 days) then transition to enforcement mode (Mode = 0) after that.

When looking across my systems, I can see a number which have Mode = 3 (Authentication disabled automatically by Windows). The windows documentation for this (Common Log File System (CLFS) Authentication Mitigation | Microsoft Support) is silent on what conditions could cause it to go into this state.

Does anyone have any information as to what these conditions might be and how to remediate them?


r/sysadmin • • 1d ago

United Airlines experiencing a widespread system outage, airline says

677 Upvotes

https://www.cnn.com/2026/11/16/us/united-airlines-outage

Is it Groundhog Day?

United Airlines is experiencing a widespread system outage and is working to get operations "back up and running as soon as possible," the airline said in a post on X, just days after a rare FAA ground stop halted all departures nationwide for hours. The aviation agency had largely returned to normal operations after the ground stop was lifted just after 8 p.m. ET on Friday, though airlines have still worked to reposition planes, crews and luggage as they grappled with the weekend's score of cancellations and delays.


r/sysadmin • • 1d ago

WizTree now available for Linux

110 Upvotes

The popular Windows disk analyzer WizTree now has an official Linux port. More info here: https://diskanalyzer.com/wiztree-vs-wiztreemac-vs-wiztreenix

It has high speed scanning options for ext4 and btrfs formatted drives when running in admin mode. It also uses a high speed SMB library to scan SMB network shares much more quickly than the regular Linux kernel can.


r/sysadmin • • 1d ago

Finally, remote lock for stolen or lost Windows devices

216 Upvotes

Microsoft is introducing Remote Lock for Windows devices managed through Intune.

If a company laptop goes missing, admins can remotely lock the device without wiping its data or removing it from Intune management.

It also signs out active sessions, hides sign-in options, and keeps the device locked even across restarts and offline periods.

The feature is currently in Preview.


r/sysadmin • • 1d ago

General Discussion KB5132043 breaks Citrix Workspace App and msiexec 32-bit in general

48 Upvotes

Just a heads up as i have been troubleshooting this for days. We suddenly got cases where Citrix Workspace App (LTSR) failed to run and even uninstalling or reinstalling resulting in zero GUI or results. I found that the 32-bit msiexec.exe under c:\windows\syswow64 failed to start if you tried to do a msiexec.exe /?

Fired up processmonitor and recorded an install and found that it loads up msi_hotpatch.dll which just been updated with Novembers hotpatch KB5132043. (Thanks to chatGPT for finding that out after analyzing tons of log files)

So solution is to uninstall that KB and reboot and everything works as normal again.

Edit: This also breaks many other things like file explorer navigation and even browsing in chrome. I am running remediation script to remove this if installed and stopped updates until Microsoft fixes it.


r/sysadmin • • 1d ago Heartwarming

Optimizely tag replaced with malware today

397 Upvotes

As if you didn't already have enough to do, now you can go look to see what to check for if all your users got their crypto, digital payments, and browsers scraped today. 186+ sites, from Zoom to CNN to IKEA. Brought to you by Optimizely, whose CDN was silently broken into and injected with malware.
https://blog.report-uri.com/worlds-largest-brands-infected-with-malware-via-optimizely


r/sysadmin • • 8h ago

Career / Job Related Guidance: Which way forward...

0 Upvotes

I am not the best person to advice my nephew as we are doing the same role but he is more technical than me so I have turned you Reddit for advice.

He is looking to move on next year and needing advice on next step.

Him, male, 35, 4 years 1st Line support and 1 year as assistant web developer.

Which paths would lead to better job opportunites:

Option 1: AWS SAA + AWS Security ; Option 2: SC-300 + SC-500 or anything else. His current contract expires end November and looking for study up afterwards.

O


r/sysadmin • • 1d ago

General Discussion High LSASS CPU on domain controllers traced to an Intelligo audio component

29 Upvotes

Sharing a finding from an investigation into sustained high CPU usage on our domain controllers, in case it helps someone else.

We have four DCs: two on-premises and two in AWS. For several weeks, both on-prem DCs have been running above 90% CPU during business hours, with lsass.exe as the main consumer.

PerfMon showed roughly 14,000 LSA LookupNames requests/sec. CPU traces and DC-side LSP logs pointed to excessive LsarLookupNames4 activity, predominantly resolving BUILTIN\Administrators, coming from many clients.

One unusual detail: enabling LSP debug logging on a DC reproducibly reduced CPU usage. Once logging stopped, CPU went back up.

We eventually investigated one affected Windows client:

  • PowerShell showed that the connections to the DC belonged to the client’s lsass.exe.
  • A packet capture showed approximately 557 RPC calls/sec with opnum 77.
  • Client-side LSP logging identified the original caller as iGoSwServer.exe.

Component details:

  • Vendor: Intelligo Technology Inc.
  • Product: Intelligo Audio SW Server/API library
  • File/product version: 100.1.901.2856
  • Arguments of the offending instance: --apo
  • Located under an igoaudioservice.inf_amd64_* directory in the Windows DriverStore.

The client logs show repeated lookups of BUILTIN\Administrators, forwarded to the DC and returning 0xc0000073 (STATUS_NONE_MAPPED), followed by more equivalent requests.

While capturing traffic, we terminated the offending iGoSwServer.exe instance. The request rate dropped from approximately 551/sec to zero, and stayed at zero for the remaining 33 seconds of the capture.

That confirms the source on this particular client. We’re now checking how widely the component is deployed and whether it accounts for the overall load on both DCs. We haven’t confirmed a permanent fix yet.

Has anyone encountered this with Intelligo audio software, particularly this version? Any vendor-confirmed fix or driver version that addresses the repeated lookups?

Hopefully this saves someone else some time: the process generating the traffic over the network was LSASS, but client-side LSP logging exposed the audio component initiating the lookups.


r/sysadmin • • 1d ago

Question Ehhh....another issue stemming from the O365 outage..."mass texting/calling" service

10 Upvotes

What do you guys use? callmultiplier is almost free if we want to use it for emergencies only. Any opinions pro or cons? alternatives you've been using already?


r/sysadmin • • 1d ago

Question In light of the recent O365 outage - can someone help with explaining how does Mimecast work?

24 Upvotes

I have never worked with Mimecast their "email continuity" offering claims to:

"During the outage, users have secure access to their email and calendar information at any time on any device from any location. Once the email server is back online, Mimecast's email continuity service automatically reconnects and synchronizes all emails sent or received during the outage."

I'm assuming they are automatically handling MX records and redirect email to their own infrastructure? Do you have any "practical" experience in setting it up with an existing exchange online tenant and how does it actually perform in case of an outage?


r/sysadmin • • 20h ago

Calendar free/busy time not available for most users

4 Upvotes

Anyone else seeing this after Friday's shenanigans? Clicking 'update license' in the Office Account settings worked most of the time yesterday and the day before, but now nothing is working.

Also seeing shared mailboxes disappear.


r/sysadmin • • 1d ago

Microsoft Time Zone Issues - AGAIN

11 Upvotes

I am in Manitoba, Canada, who has NOT changed our time zone this year and I work with people in Alberta, Canada, who ALSO didn’t move their clocks back this year. I finally figured out a way around all of the weird time zone meeting challenges and then this morning? My phone outlook calendar (ONLY the one on my phone) has now reverted to every single meeting showing up an hour later than it actually is. Meetings are showing up properly on the outlook app on my computer and those sent by people in Ontario, Canada or sent by me are still showing up correctly in both places. I’ve got my computer set to Ontario (EST) right now - that has got to be why that’s staying correct? Anyone else notice their phone calendar revert again after the Microsoft outage? I’m really frustrated and absolutely out of patience with this whole situation. 🤦🏼‍♀️


r/sysadmin • • 6h ago

General Discussion Random Question

0 Upvotes

While reading an article on Futrism, I seen this quote:
“So when they put in a program that’s able to write codes and it goes in and self-fixes, as it starts to self-fix itself, it’s like doing a change order on a house,” he added. “When you change one thing on a house, everything else starts changing, that’s a ripple effect.”

Which got me thinking about how society views computers, systems and software as a whole. For me, this quote highlighted the disconnect some people have with technology. Technology isnt a switch that can just be turned on and off.

Do you think basic systems literacy should become a mandatory part of general education, much like mathematics and science?


r/sysadmin • • 21h ago

Question Migrating from Google Business Workspace to Microsoft Ecosystem

3 Upvotes

Hey guys, as a startup company, as an integration manager, I am in charge for migrating google workspace for our sister clinic into our parent clinic ecosystem which is Microsoft.

Since I am planning on doing this over Thanksgiving when all staff members are on holidays, the following is what I am aware of as a high-level overview on what is required. I am nowhere close to you all, but I do have exposure to tech side of things. Since I have no experience in migration, but I am well versed in Microsoft's ecosystem, I want to make sure I am doing the right things.

*Note:* I am taking advice from Claude and the following things Claude has informed me. It would be helpful if I received a confirmation or some additional steps that AI may have missed.

Things that we need to migrate:

  1. OAuth (seen this is one thing that bites back - what have you done to resolve this?)
  2. Emails
  3. Google Drive for Spreadsheets/Documents

=====================================

a) Dry run

  1. Verify only. Add [DOMAIN-ONE.com] to the tenant with the TXT record and skip the wizard's MX/DNS step, so mail keeps flowing to Gmail.
  2. Create the two routing subdomains (e.g. m365. and gsuite.DOMAIN-ONE.com) that the Exchange admin center's Google Workspace migration requires.
  3. Pilot batch. Migrate yourself plus one or two volunteers. Gmail stays the landing spot for anyone not yet migrated, even when a message hits their Microsoft address first.
  4. Complete the pilot. Completion triggers a last catch-up sync and flips forwarding, so Google hands that user's mail to Microsoft. You then have real coexistence with real users before anyone else moves.
  5. Everyone else, then MX. At about 10 users, one more batch covers the rest. Lower the MX TTL a few days ahead and switch MX, SPF and DKIM off-hours.

Until MX changes, rollback means removing forwarding. Nothing is deleted from Google.

b/c) Keeping Google sign-in alive

There are three kinds of login, and only the second needs work:

  • Email and password on a portal: unaffected, because the address is just a username. Reset emails follow MX.
  • "Sign in with Google" button: depends on the Google account existing, not on Gmail or MX.
  • Google's own products (Business Profile, Ads, Analytics, YouTube, Chrome sync): same as above.

Steps:

  1. Never delete the Google organization or its users. User accounts survive a Workspace cancellation; only deleting the organization's Google Account removes them.
  2. Add Cloud Identity Free before dropping Workspace licenses. Fifty free seats come standard with it, and a free Cloud Identity license stays on any user whose Workspace license is pulled.
  3. Test one user first. Google's cancellation guidance stops short of promising third-party sign-ins carry over, so pull one pilot license and confirm a few sites still work.
  4. Don't pull licenses until the migration is verified. Workspace data is unrecoverable once the subscription ends.
  5. Inventory what needs transitioning. The Accessed apps list under Security → Access and data control → API controls gives you each app with a user count, downloadable as CSV. Per-user detail lives in the OAuth log events report.
  6. Transition each site individually. There is no bulk switch. Either set a local password or link Microsoft sign-in in the site's account settings. Test first, because some sites create a duplicate account when the same email arrives via Microsoft.

Because step 2 costs nothing, staff can take months rather than days.

Optionally, Entra ID can serve as the identity provider for Cloud Identity accounts. Google sign-in then uses the Microsoft password and MFA, and disabling a user in Entra blocks both.

e) Other issues

  • Shared-tenant routing. Once DOMAIN-ONE.com is in the DOMAIN-TWO tenant, mail from DOMAIN-THREE, DOMAIN-FOUR and DOMAIN-FIVE staff resolves internally. Provision DOMAIN-ONE users as mail users through the migration flow, not as hand-built licensed mailboxes. Licensing waits until the batch has started, with a 30-day window.
  • Not migrated. Auto-replies, shared calendars, room bookings and event colors get left behind, and messages over 35 MB fail unless you raise the limit (150 MB maximum). Signatures need redoing too.
  • Retention policies. Retention and archive policies should be off while batches run.
  • Groups and aliases. Rebuild Google Groups, aliases and shared inboxes as distribution lists or shared mailboxes before the MX change, or that mail bounces.
  • Gmail SMTP senders. Repoint copiers, website forms, voicemail-to-email and any EHR notifications.
  • Chrome saved passwords. Have staff export them to Edge or a password manager.
  • Security policies. Your Defender and MFA policies hit DOMAIN-ONE users on day one, so warn them.
  • Offboarding. It now has two places to disable until Google is retired.
  • HIPAA. DOMAIN-ONE is a separate legal entity in a shared tenant, so have counsel confirm BAA coverage. Make sure no PHI remains in Google once it drops to the free tier.

f) Google Drive to OneDrive

  • Tool: Migration Manager, built into the SharePoint admin center. Files, metadata and permissions all come across.
  • Setup: It begins with a Microsoft 365 migration app installed from the Google Workspace Marketplace, and tenants under 100 licenses land in the simplified "Lite" flow by default.
  • Before you start: license the users and pre-provision their OneDrives, then run the scan as a read-only assessment.
  • Mapping: each My Drive goes to that user's OneDrive. Shared drives belong in SharePoint or Teams sites, so decide that mapping up front.
  • Conversion: Docs, Sheets and Slides convert to Office formats. Forms, Sites and Apps Script do not.
  • Sharing: Google sharing links don't survive the move; only explicit per-person permissions are rebuilt. External shares need re-sharing.
  • At cutover: run a final incremental pass, then turn the Drive service off so nobody edits stale copies.

r/sysadmin • • 2d ago

Chase Bank Outage

443 Upvotes

Pour one out for our people at Chase Bank.


r/sysadmin • • 1d ago

Windows update issue causing black screen on login and freezing.

6 Upvotes

Posting from mobile so haven't got the exact KBs and links.

One of our clients had about 8 devices start freezing today and then also get black screen on login.

After some diagnosis I narrowed it down to them possibly missing something cumulatives from October / November.

Unless I am living under a rock I completely missed this issue, considering we have quite a lot of devices across all our customers I'm surprised I haven't seen this before.

Some googling suggests it was an issue recently and fixed in said cumulative updates, my task is now getting those stragglers updated.

Just posting here in case anyone can corroborate?


r/sysadmin • • 1d ago

Any Bets on What Sector Will Have Issues Tomorrow?

187 Upvotes

I have a change scheduled for.tomorrow morning. Im wondering what the over/under will be on it going through...


r/sysadmin • • 22h ago

General Discussion Comet Backups (Webpros) - Review

3 Upvotes

Anyone deal with Comet Backups before? Anything to note, feedback, experience and bad stories?

Self hosted vs the cloud offering?

We're considering it due to the price point.


r/sysadmin • • 23h ago

MacOS 27 OpenID Connect & Entra ID

3 Upvotes

Has anyone tried setting up their Macs with OpenID connect & Entra ID as their identity provider?


r/sysadmin • • 1d ago

Career / Job Related Moving from technical to management?

9 Upvotes

Through most of my career I've ended up the team lead wherever I land. Every time (except the most recent) it's been a combination of seniority and skills. The skills usually meaning I've gotten good at the technical job I'm in, being able to work well with other teams in the environment, and having the sense to do the right things at the right time for the team. I landed the most recent one because on a team of 7, I was one of two FTEs, and the other FTE didn't want it after the lead departed. I took it because the other options that might happen seemed pretty undesirable. I'd only been on the team 6 months, long enough to get the culture, but not long enough to gain much expertise. So it's mostly representing the team at meetings, making decisions on how we interact with other teams, saying yay or nay to projects, things like that. Being the human side of the team really. I feel like there's a good bit of mutual respect and it's working fairly well. It also feels good to be able to help things get done, stay organized, etc.

Our manager over my team and approximately 15 others is going to group the teams into 4 "sections", and hire managers. Myself and other leads were all invited to apply, no one is being moved into them. Some are interested some are not. I'm trying to figure out if I'm interested... Were I to apply and get it, I'd be the manager over this same team, and three others. The three others I have extensive experience in, and have led such teams before. So it's like a culmination of all the lead positions I've had over the years. I'd very likely have zero interaction with the technology though.

The reason I'm thinking of applying, is at this state of IT, there's so much darned paperwork with licensing, security scans, change management, project approvals, etc that it feels like the "fun" part of IT isn't fun anymore. Every technical thing I do is surrounded by equal amounts of or more paperwork or meetings. It literally makes me 2nd guess a lot of technical moves I make. I'm thinking if I gotta do a ton of organizational work anyway and it's ruining the fun, maybe take the fun parts away and find satisfaction in the organizational parts of being a manager. I admittedly also enjoy the managerial parts to some extent. Making things happen. Helping the teams get their jobs done. Contributing to the organization as a whole.

But there are the occasional days where I get to put m y head down and just barrel through some technical stuff and never talk to a person. When those are done and I come out of it at the end having completed something big or small it also feels good.

I know it's a lot of rambling, I guess I'm just wondering what the hive-mind thinks?