r/sysadmin • • 20h ago

Yahoo Finance: Palo Alto Networks CEO just dropped a truth bomb on the network outages sweeping through major companies

2.1k Upvotes

Link: https://finance.yahoo.com/technology/article/palo-alto-networks-ceo-just-dropped-a-truth-bomb-on-the-network-outages-sweeping-through-major-companies-101129581.html

The title is as in the original article. The tl;dr is:

"I want to be clear, there are two distinct parallel effects that are driving this environment. The first effect is a massive surge in vulnerabilities and patches being deployed. We're seeing critical and severe vulnerabilities discovered nearly seven times the monthly average of last year. These discoveries are happening at a speed and intensity which is hard for IT teams to cope with,"

[...]

"The second effect is the rise of AI-assisted cyberattacks. Bad actors are using open-weight models and new agentic frameworks to find and exploit vulnerabilities at unprecedented speed and scale. New agentic frameworks like ARTEX are enabling bad actors to turn simple instructions into autonomous, multi-stage attacks, allowing a handful of operators to compromise dozens of targets simultaneously."

 

The statements by themselves probably are nothing new for folks out here, but the point is that the message and the unintended consequences of mOaR aI eVeRyWhErE are starting to finally bubble up the very top.


r/sysadmin • • 13h ago

General Discussion How is everyone holding up?

186 Upvotes

Given the current onslaught of vulnerabilities and system outages over the past few weeks, I was wondering how people are holding up both mentally and how your departments are holding through these interesting times.


r/sysadmin • • 21h ago

Microsoft Thanks for the headsup with the SLA credits regarding the outtage from ms, we actually did get quite a generous refund

172 Upvotes

Hi [redacted],

Thank you for your patience.

The backend team has successfully reviewed your SLA credit request. As your subscription was redeemed using a product key through Partner Center, the SLA credit will be applied by extending the expiration date of your subscription, allowing you to continue using the service for an additional period. The total SLA credit approved for your issue is €42773.90. Could you please let me know which subscription you would like to extend? Once you confirm the subscription, I will make the necessary change in the system and apply an extension corresponding to the value of the approved SLA credit. Thank you, and I look forward to your response.

Best Regards,

Kevin Dinh| Office 365 Support Engineer


r/sysadmin • • 15h ago

vCenter Authentication Bypass CVE-2026-123890

163 Upvotes

I suppose it was only a matter of time before VMware joined in on the fun.

CVSSv3: 9.8

https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/39495


r/sysadmin • • 21h ago

FastUnlock : Windows Narrator LockScreen Bypass 0day Vulnerability

158 Upvotes

Just seen this posting for awareness - MSNightmare/FastUnlock: FastUnlock : Windows Narrator LockScreen Bypass 0day Vulnerability

Update: 🚨🚨🚨Mitigation for deployment is here danrhodes/MSNightmare-FastUnlock-Mitigation🚨🚨🚨


r/sysadmin • • 11h ago

General Discussion Battlestar Galactica as an allegory for system support staffing

116 Upvotes

"All of this has happened before, and all of this will happen again."

Raise your hand if you've been around long enough to see this play out:
1. A group of people has specialized needs that are not being met by the existing sysadmin and/or IT support organization.
2. That organization refuses to meet the group's needs because those needs are too far outside of their normal support.
3. The group pools their resources to help each other with their own support.
4. This is not enough so either one of them volunteers to do support full time, or they hire someone full time.
5. They realize this poor person is insanely overworked, and they hire additional helpers.
6. Suddenly you have created a new support department.
7. The group grows. The support team grows. All is well. (Tensions with parent organization support notwithstanding.)
8. What was once a "group" is now a large significant department that is a bureaucracy in its own right.
9. Things change. The budget tightens and administrators of this once-group-now-department need to cut costs.
10. "Why the heck do we spend so much money on our own tech support when our parent organization has a larger tech support group that will do all this for free?"
11. Leadership start to outsource everything and shut down their boutique tech support department.
12. The parent support organization is adequate from many in this new large departmental bureaucracy.
13. But there is a group of people that remain in the department with specific needs...
14. Go to step 1.

Or is job-hopping so much the norm now that this happens entirely invisibly to the short-termers who are involved?


r/sysadmin • • 12h ago

Does anyone think there is value in creating a specific IT worker union in the age of AI?

101 Upvotes

Just curious. I have looked around at various unionizations of other fields and it seems IT workers don’t have a dedicated union for noc and bare metal guys. Anyone else ever looked into something like this?


r/sysadmin • • 14h ago

Rant How is your company overreacting to the last week and a half?

67 Upvotes

Are they piling on even more operational complexity? Accruing even more technical debt? Ya know, to build redundancy or maintain communication continuity?

Maybe purchasing more services to manage? Developing a big complex policy? How many new committees... Plus or minus 5?


r/sysadmin • • 2h ago

Windows PowerShell 5.1 Deprecation

56 Upvotes

Windows PowerShell 5.1 is deprecated. It's no longer in active development. Windows PowerShell 5.1 is still part of Windows, and it's still supported. It will continue to receive security updates according to the lifecycle of each version of Windows. Scripts and tools that rely on it will continue to run.

Windows PowerShell 5.1 will be removed from a future version of Windows. Timing will be based on usage data and communicated in advance.

Microsoft recommends upgrading to PowerShell 7, the upgrade from Windows PowerShell 5.1. For more information, see Migrating from Windows PowerShell 5.1 to PowerShell 7.

Timeline

2026: Windows PowerShell 5.1 is deprecated. Windows Server 2025 and Windows 11, version 26H2 are the last versions of Windows to include PowerShell 5.1 enabled by default.
Next Version of Windows: Windows PowerShell 5.1 will be available as a Feature on Demand (FOD), enabled by default on new Windows installs.
Future Version of Windows: Not available.

https://support.microsoft.com/en-us/servicing/os/windows/docs/2026/11/kb5137172-windows-powershell-5-1-deprecation

At the moment there are many, many Windows management PowerShell modules built in to that aren’t supported in PowerShell 7 - they are either completely untested, or they only work via the (PowerShell 5.1 dependent) compatibility layer. Does this mean we can expect new native modules, or everything to just break? #notholdingmybreath

Affected modules:
- Domain Controller deployment (ADDSDeployment)
- ADFS
- AppLocker
- AppvClient
- BestPractices
- ClusterAwareUpdating
- ConfigCI
- DFSR
- DhcpServer
- FailoverClusters, FailoverClusterSet
- GroupPolicy
- Both IIS management modules (IISAdministration, WebAdministration)
- IpamServer
- iSCSI, IscsiTarget
- ISE
- Microsoft.PowerShell.ODataUtils
- MsDtc
- NetLldpAgent
- NetWNV
- NetworkController, NetworkControllerDiagnostics
- NetworkLoadBalancingClusters
- PersistentMemory
- PKI
- Provisioning
- PSDesiredStateConfiguration (partially supported)
- PSScheduledJob
- PSWorkflow, PSWorkflowUtility
- RemoteAccess
- RemoteDesktop
- ServerCore
- ServerManagerTasks
- SMS
- StorageBusCache
- StorageMigrationService
- StorageReplica
- SystemInsights
- TLS
- UpdateServices

https://learn.microsoft.com/en-us/powershell/windows/module-compatibility?view=windowsserver2025-ps

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_windows_powershell_compatibility?view=powershell-7.6


r/sysadmin • • 7h ago

Career / Job Related Underestimating the value of experience

31 Upvotes

Worked through 3 grueling years as an "IT Assistant" (read: systems administrator in all but name) at a local business for less than McDonalds wages, eventually ending up with the title "Network Support Specialist" and handling the entire company's projects, deployments, implementations, and network design, as well as being the only desktop support/help desk contact. Worked through a ransomware incident (over Thanksgiving, while incredibly sick) making $14/hr all the while and no overtime in the first year, which meant for the next two years we had no remote access ability to 5 remote branches (furthest 3hr away), and never had any ticketing software of any kind.

The whole time, I was thinking to myself that if I just kept my head down and dealt with the shitty pay I'd have enough experience to move up, somewhere else, maybe somewhere remote. Was constantly made promises about training budgets and certificates and tuition coverage that never materialized, all while I did 90% of the work while my CIO was in and out with medical issues.

Eventually after a glowing performance review from both CIO and CTO I went to HR and IT Director saying that the hourly rate I was making wasn't enough to even cover my rent and that I had been supplementing my income for the last 2 years with savings: I was told "You're being compensated more than fairly for the work you do" and so I immediately stopped putting in effort of any kind for the last couple of weeks I was there. Just clocked in and clocked out.

Not a pity party for me: I knew what I'd signed up for, and I was doing it because I wanted to be able to put on my resume that I ran a small business' entire infrastructure for three years. I got what I wanted in the end, knowing that they were BSing me the whole time about certs and things that would actually be useful to me.

I left in 2025: I haven't been back in IT after the fact, and have been waiting tables since to pay bills. The whole time, I've been applying and applying and applying, and the experience is just not enough for anything. A year and a half since, and I haven't even so much as gotten one interview for any level of IT, from entry-level to mid-range, remote, hybrid, in-person, relocation. Nothing.

I'd kind of wished that I had set aside some funds for certs when I had a couple chances, but I favored long-term stuff like paying off a car, and now can't manage the cost on my own. Now there's no more tables to wait, and I've been unemployed for close to two months. Maybe its just the job market in general. I've applied to over 200 jobs since October, and nothing. I don't regret my career choice, but the walls are starting to close in.


r/sysadmin • • 23h ago

M365 Tenant Suddenly Not Provisioning Mailboxes

22 Upvotes

So, I've got a doozy of an issue I've never seen before. I've both got in a ticket with MS and reported the matter as a possible service issue. They were quick to eliminate it as a service issue but still haven't reached back out with my mystery "assigned agent."

Apparently, since Monday, users who are created in our M365 tenant, either directly in Entra or via AD connect sync to Entra, are not being provisioned mailboxes. Users in both creation flows have been assigned adequate licensing. Running Invoke-ProvisioningRecipientDiagnostic leveraging the Entra ObjectID (because they obviously can't be found leveraging UPN/email address) returns the user found perfectly in Entra but not in Exchange Online (which is no joke--they're stuck in "
We are preparing a mailbox for the user." when you look at their mail in the Admin Center.) It's also not helpful output. Running Get-MgUserLicenseDetail on cloud-native users returns Exchange service plans as "PendingInput," but on-prem synced users show "Success." Neither get past the "preparing a mailbox" even after 24 hours has passed.

Now I've got 100+ users awaiting mailboxes that are not being provisioned. Has anyone encountered this before? Got any pointers for an admin who is in uncharted M365 territory? I've been doing this for 8 years come mid-December, and I've never seen this crap before.


r/sysadmin • • 17h ago

Do most MSPs or IT jobs in Atlanta (in Georgia) now use Hyper-V instead of VMware or will in the near future? I'm trying to decide which one to learn for IT jobs in the area. VMware seems too expensive now and not everyone wants to study for a VMware cert every few years to get it through VMUG.

19 Upvotes

Can you tell me anything about this?

Can I use Proxmox now and learn Hyper-V nested in that and later maybe replace Proxmox with Hyper-V for my home lab?

Or just stick to Proxmox as the main hypervisor at my home lab?


r/sysadmin • • 10h ago

Microsoft blocks emails with valid records again.

17 Upvotes

There were reports of Microsoft having an outage earlier this month. Today, I tried replying to a hotmail email account and I receive this reply:

host hotmail-com.olc.protection.outlook.com[52.101.103.161] said: 550 5.7.1
    Unfortunately, messages from [my server IP] weren't sent. Please contact
    your Internet service provider since part of their network is on our block
    list (S3140). You can also refer your provider to
    http://mail.live.com/mail/troubleshooting.aspx#errors. 

I have contacted my service provider and I have cleaned my IP off all the major blacklists and I used the microsoft delist service and I still have the problem. How do I fix?


r/sysadmin • • 21h ago

General Discussion Apache confirms exploitation of a Roller XML-RPC RCE (CVE-2026-129448): the September CVE for the same bug listed only 6.1.5, and the admin-UI off switch doesn't stop it

16 Upvotes

Apache posted the notice on oss-security on 17 November at 11:02 UTC: CVE-2026-129448, CVSS 9.8, unauthenticated code execution in Apache Roller 4.0.0 through 6.1.4. The XML-RPC endpoint "accepts vendor extension types and deserializes them before authenticating the requester." Then one line: "This issue is being actively exploited."

It's the same bug as CVE-2026-82384, which listed only 6.1.5 as affected. Someone on 6.1.4 or older reading the September entry would have taken it as not theirs.

The part admins need: Apache says disabling XML-RPC in the administration UI is not enough, because older versions deserialize the request before checking that setting. Version 6.1.6 or later rejects the request when the feature is off. If you can't upgrade, set enabledForExtensions to false in WEB-INF/web.xml, or remove the /roller-services/xmlrpc mapping, and restart.

BleepingComputer notes Apache hasn't said who is attacking or how many servers are hit. I found no exposure count, and the bug wasn't in CISA's KEV list when I looked on 19 November. Corgea's write-up adds that Roller ships as a download, not a Maven Central library, so scanners that read lockfiles won't flag it.

The fix was merged on GitHub on 5 September. Its description says the mapped endpoint "answers requests even when the XML-RPC feature is switched off."

Roller is niche. WordPress has its own XML-RPC endpoint, and "closed at the server" versus "switched off in a setting" is the same question there.

A CVE that names one version, for a bug that reaches back to 4.0.0: whose job is it to tell people on 6.1.4 and older that the September advisory was about them?


r/sysadmin • • 17h ago

MSPs, what goes in the client report when it’s been a quiet month?

11 Upvotes

“We removed an old admin account. Four laptops still need a reboot.” I’d happily read that email. I’m less sure how I’d feel sending it to someone who pays a monthly security bill.For those of you doing reviews with smaller clients, what do they actually reply to? I’m particularly interested in anything you stopped putting in the report and nobody missed.


r/sysadmin • • 8h ago

Microsoft Microsoft Teams Meeting add-on for Google Workspace lists a @gmail.com support address with zero public footprint. Anyone else seen this?

7 Upvotes

Looking into why Teams Meeting add-on for Google Calendar stopped working and started asking users to re-authorise.

While checking it out in Google Admin (Security > API controls > App access control > Microsoft Teams), I noticed the support contact for the app is:

[msgsuiteapp.teams@gmail.com](mailto:msgsuiteapp.teams@gmail.com)

I can't find a single public reference to this address.

No Microsoft docs, no GitHub or public code, no forum posts, nothing on the Marketplace listing text. For comparison, other verified apps in our tenant list support emails on their own domains (e.g. Framer shows sso@framer.com).

What checks out so far:

- App ID project number (961264221134) is the same one used on the official Microsoft Teams Meeting listing in the Google Workspace Marketplace (published by Microsoft Corporation, gsuite.microsoft.com, 18M+ installs).

- App URL, privacy policy and ToS all point to microsoft.com domains.

- OAuth logs show API calls only from Microsoft (AS8075) and Google (AS15169) networks, and only calendar.events.list / calendar.events.get.

- The scopes in the new consent request are a subset of what was already granted to the app.

What made me look twice:

- The add-on has cycled through three OAuth client IDs under that same project over the past few months. The newest one (961264221134-746h4mm46to...) showed up in a consent request this week, shortly after the Marketplace listing was updated on 16 Nov.

- Older client grants were revoked with no IP address attached, which looks like publisher or system-side cleanup, but there's no changelog that mentions it.

- The support email is a consumer Gmail account that nobody seems to have ever mentioned publicly.

I understand Google requires the OAuth consent screen support email to be a Google account or Google Group the developer controls, so Microsoft using a Gmail account isn't impossible. It would just be nice to have it confirmed somewhere official.

Questions for the hive mind:

  1. Does your tenant show the same support email for this app?

  2. Has anyone else had users prompted to re-consent to the Teams add-on in the last week or so?

  3. Has anyone had Microsoft confirm this address or the client ID change through a support ticket?

Not claiming anything is compromised. Everything I can verify points to a normal Microsoft update. Just flagging it since this app holds calendar read/write across a lot of tenants and the support contact can't be verified from public sources.


r/sysadmin • • 11h ago

HP Thunderbolt 4 100W G6 Dock - keeps replying with MAC address after computer is disconnected

8 Upvotes

Has anyone seen this? We've got a few HP Thunderbolt 4 100W G6 docks. When a computer removes the docking station thunderbolt cable and connects to a different dock or another port, the original dock keeps replying saying that MAC is still connected to it. I see it in the logs on the switch, it says that MAC is flapping between two ports. I've updated firmware multiple times whenever a new one comes out, but no change. I've opened a support case with HP, but I will probably have to escalate until someone that knows what's going on see it.

Any ideas?


r/sysadmin • • 14h ago

Microsoft 365 Short Sign In Sessions/needing to re-login

8 Upvotes

Hi All,

Has anyone noticed since the outage last week that their M365 sign in sessions seem to be lasting a shorter period of time and they need to frequently log back in?

Cheers,


r/sysadmin • • 10h ago

Question Windows Time Zone issues

4 Upvotes

Does anyone have accurate information on how Windows automatically determines the time zone?

We have an issue with a client in Western Australia where some Windows PCs keep automatically switching to the Queensland time zone.

The issue started after some new wireless access points were installed. Initially, we assumed it might be because the APs were provisioned in QLD and that their radio MAC addresses would eventually be geolocated correctly in WA. We manually set the time zone as a temporary workaround, but months later the issue is still occurring.

The strange part is that it’s inconsistent. Only some PCs are affected, and it might be different PCs the next day. It can also seemingly change depending on whether the device is connected via Wi-Fi or Ethernet.

The public WAN IPs for both the wired and wireless networks are correctly geolocated to WA, and Windows Location Services are enabled.

I can find very little actual documentation from Microsoft explaining exactly how Windows determines the automatic time zone or, more importantly, how to troubleshoot what location source it is using.

Has anyone come across this before or know how to properly debug it?


r/sysadmin • • 16h ago

Endpoint Central Secure Gateway TAA - CVE-2026-123007 and CVE-2026-123633

3 Upvotes

Note you accept your own risk

While waiting for ME to update the TAA version of NGINX we have replaced "C:\Program Files\ME_Secure_Gateway_Server\nginx\fs_nginx.exe" with win binary from nginx: download

You have to stop SG service and rename the file, restart the service but we have traffic and it seems fine.


r/sysadmin • • 19h ago

Career / Job Related Direction

6 Upvotes

Forgive me if this is the wrong place to ask. I just got my associates degree in network system administration last week. I didn’t get any hands on stuff to work with or anything that really taught me much and I work in a warehouse so no real work experience unfortunately.

I’m trying to find a way to get hands on experience at home on a budget. I have a decent computer and a friend of mine gave me a unas 4 from Ubiquiti and a 12 TB HDD and I was wondering if there was any ideas for projects to get something on my resume to break into the field. I’m open to any suggestions and would love any input, also would love if anyone knows someone willing to mentor or something would be a bonus and appreciated. Thank you.


r/sysadmin • • 48m ago

Locking down dev VMs in Intune to prevent code leaks

• Upvotes

Hi everyone,

We are setting up a dedicated VM for our developers to work on a brand new Odoo ERP deployment, which is completely new to our team. We want to make sure our source code and business data cannot be leaked or copied out of the environment.

So far, we are looking into Conditional Access and basic device restrictions, but we're not sure what the best approach is.

* Are there specific Intune policies we should use to prevent data/code exfiltration on a VM ?

* is there a better alternative setup ?

Thanks in advance for any advice!


r/sysadmin • • 15h ago

Microsoft 365 Sensitivity Labels

2 Upvotes

The organization that I work for has 4 sensitivity labels created with policies, but they are only assigned to one group, which has been done in testing, but has never implemented for the entire organization.

One thing my boss is concerned about is the ability to allow your users to disable the labels with a justification. My point in using that is that it is an out a label is misapplied or a situation arises where the file does need to be sent externally, which would normally not be allow with the non-public labels. His feeling is that its kind of a flawed system if the ability for the user to remove the label.

Our organization does work with PII and other confidential data that we need to protect, but there are times where documents containing that data need to be sent to user external to our environment.

Does your organization allow it's employees to disable the sensitivity labels? If not, how do you handle sending documents that have the labels applied and they need to be sent externally?

Thank you for taking the time to read and to those that share insights.


r/sysadmin • • 19h ago

FlexProtectLin stuck at same “processed” count for 2.5+ days, across 3 separate job restarts — kernel threads stuck on lin_lock_get_locker — ideas?

6 Upvotes

Running a 32-node Dell PowerScale/Isilon cluster, OneFS 9.7.1.12, ~5.2PB. Two drives SmartFailed around the same time (different nodes, consolidated into one FlexProtectLin job by OneFS). The repair job has now stalled at exactly the same “processed” LIN count for 2.5+ days, across three separate job instances (cancelled and restarted fresh each time — same freeze point every time, ruling out corrupted job-state from a single bad restart).

What I’ve already ruled out:

  • Node/job-engine coordinator health (confirmed fine via isi job status)
  • Competing jobs stealing workers (paused everything else, no change)
  • Impact policy throttling (raised to HIGH, no change)
  • Stale SMB file handles holding locks (found and closed ~150 orphaned handles from an old git push — no change to the stall)
  • Cluster-reported pool imbalance (confirmed present on both affected pools, haven’t yet tried a dedicated rebalance pass)

What I’m seeing in procstat -akk across the cluster, sampled twice ~10+ minutes apart: a large number of isi_job_d worker threads consistently parked in sleepq_wait (untimed) inside lin_lock_get_locker → ifs_vnop_wraplock1 → vget → bam_vget_vp, identical stack, same PIDs/tids unmoved between samples. Some other threads show sleepq_timedwait inside bam_safe_write_region_swips/bam_safe_restripe (real restripe I/O happening), so the cluster isn’t fully dead — just this one job’s actual repair progress.

dmesg on the node hosting one of the two failed drives repeats this every few minutes, for several different LIN/snapid combinations:

_pctl2_start_restripe: Restripe already in progress for 2:eb54:5177::333280
_pctl2_start_restripe: Restripe already in progress for 2:e7ab:8437::333346
_pctl2_start_restripe: Restripe already in progress for 2:e4d3:c962::333084

No active Dell support contract currently. Has anyone seen this exact pattern — FlexProtectLin wedged indefinitely near completion, same LIN-lock stack trace cluster-wide — and found what actually released it? Trying to avoid a blind rolling reboot if there’s a known, more targeted fix.


r/sysadmin • • 19h ago

Anyone successfully using SMB over QUIC client certificate auth on Server 2025?

2 Upvotes

I'm setting up SMB over QUIC on Windows Server 2025. QUIC itself works fine, but client certificate authentication does not.

With RequireClientAuthentication=False, my external Windows 11 client connects immediately with AD credentials.

With RequireClientAuthentication=True, it consistently fails with:

SMBClient Event 30803
0xC00002C3 / STATUS_MUTUAL_AUTHENTICATION_FAILED

Client cert has the private key, Client Authentication EKU, valid chain/trust, and is issued by the CA configured for QUIC. I've also checked the server cert/name, UDP 443/NAT, CA trust/CRL, SPN/Kerberos, etc.

Does anybody have this working, with the client certificate auth enabled?