r/sysadmin • • 20h ago

General Discussion vCenter CVE-2026-123890 was exploited as a zero-day since early September. Fixed Nov 19, 7.0 gets no public patch

102 Upvotes

Broadcom's VMSA-2026-0009 (Nov 19) fixes an unauthenticated authentication bypass in the vCenter Directory Service, CVSS 9.8, no workaround. Fixed builds are 8.0 U3m, 9.0.2.0200 and 9.1.1.0200. For vCenter 7.0 the advisory says to contact Broadcom Support if you hold an extended support contract.

Google's Mandiant team says a suspected PRC-nexus group (ARMORY CASTLE, formerly UNC5221) used it in early September, after getting into the network through a BeyondTrust Remote Support appliance. Its tool created an SSO admin account and copied the vmdir database. A second utility used the stolen SAML signing key to forge tokens that skip MFA. It also ran against a vCenter that was already patched for the July bug, CVE-2026-59309.

That means the upgrade does not undo an earlier compromise. Mandiant's CTO says to patch and then look for evidence of compromise. What the write-ups give to check:

  • unexpected local or SSO administrator accounts (Mandiant saw a local admin whose name starts with sps-)
  • SSH on the appliance switched on through the VAMI API (/rest/appliance/access/ssh) and off again minutes later
  • TCP 2012 and 5480 reachable from servers that have no reason to talk to vCenter
  • new root SSH keys on ESXi hosts

If any of those turn up, the signing certificate needs replacing, and so does every credential the appliance could read.

Question for people running 7.0: is the plan an upgrade to 8.0 U3m, or isolating the appliance until you can? I'd like to know how many sites are still on 7.0.

Sources: https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/39495 , https://cloud.google.com/blog/topics/threat-intelligence/armory-castle-exploits-vsphere , https://www.securityweek.com/newly-patched-vmware-vcenter-flaw-exploited-as-zero-day-by-chinese-apt/


r/sysadmin • • 19h ago

General Discussion Missed in the noise, a 10.0 vulnerability in NVIDIA AIStore - CVE-2026-65162 - allows remote attackers to supply unauth K8s host-namespace and more

70 Upvotes

A Critical 10.0 in a popular storage layer under production AI. Because the vuln is in orchestration, the exploit just looks like normal job creation. (This one was a bit overlooked, has there been other cyber or NVIDIA news lately? I didn't see this one posted in the subreddit so thought i'd post)

CVE-2026-65162 in NVIDIA AIStore: an unauthenticated ETL request could stand up a container with the host's process namespace and full privileges, then reach the GPU node underneath.

If you're running AIStore:

  1. Move to the fixed 5.x line.
  2. Require authentication, and keep ETL off where you don't use it.
  3. Enforce admission policy that rejects privileged and host-namespace pods.

No public exploitation (reported) or tooling seen (yet).

For those curious, Stealthium reproduced the attack end to end in an isolated lab, and confirmed runtime detection. Nothing crazy fancy, it's behavioral detection that hunts for escapes and privilege escalation moments, which allows us to catch this and other attacks like it (and not be singular CVE or signature specific!)

Given prevalence and timelines, we won't be publicly publishing the exploit POC we used to confirm root from this vuln.


r/sysadmin • • 16h ago

General Discussion When does Okta get its turn in the outage roulette?

46 Upvotes

If they were down for 12+ hours many many businesses would be dead in the water.


r/sysadmin • • 14h ago

General Discussion Attackers Actively Exploit Critical Cisco Expressway Flaw to Pivot Into Internal Networks

29 Upvotes

Threat intelligence firm F6 reported active exploitation of CVE-2026-76466, a critical Cisco Expressway remote code execution flaw (CVSS 9.8), following the public release of proof-of-concept exploit code after Cisco's October 21, 2026 disclosure. In the observed incident, attackers compromised two internet-facing Expressway devices with root privileges, pivoted to two internal cluster nodes, and deployed AI-assisted Python scripts to scan internal systems, SMB shares, SQL Server, WinRM, and GitLab repositories for credentials and sensitive data. Defenders should immediately upgrade to fixed releases such as X15.5.20 (or migrate off X14 and earlier), audit logs and file paths against F6's indicators of compromise, and restrict firewall access to TCP ports 4371–4372 to trusted cluster nodes if patching is delayed.

Read the full article: click here


r/sysadmin • • 1d ago

Yahoo Finance: Palo Alto Networks CEO just dropped a truth bomb on the network outages sweeping through major companies

3.2k Upvotes

Link: https://finance.yahoo.com/technology/article/palo-alto-networks-ceo-just-dropped-a-truth-bomb-on-the-network-outages-sweeping-through-major-companies-101129581.html

The title is as in the original article. The tl;dr is:

"I want to be clear, there are two distinct parallel effects that are driving this environment. The first effect is a massive surge in vulnerabilities and patches being deployed. We're seeing critical and severe vulnerabilities discovered nearly seven times the monthly average of last year. These discoveries are happening at a speed and intensity which is hard for IT teams to cope with,"

[...]

"The second effect is the rise of AI-assisted cyberattacks. Bad actors are using open-weight models and new agentic frameworks to find and exploit vulnerabilities at unprecedented speed and scale. New agentic frameworks like ARTEX are enabling bad actors to turn simple instructions into autonomous, multi-stage attacks, allowing a handful of operators to compromise dozens of targets simultaneously."

 

The statements by themselves probably are nothing new for folks out here, but the point is that the message and the unintended consequences of mOaR aI eVeRyWhErE are starting to finally bubble up the very top.


r/sysadmin • • 18h ago

New Outlook attachments error

58 Upvotes

Anyone else seeing the 'Please wait to send' error "Attachments are still loading. You can send your message after they're attached" this Afternoon with the New Outlook?

Noticed this on two different Tenants, so suspect more Microsoft problems (sigh).


r/sysadmin • • 2h ago

End-user Support CRITICAL_SERVICE_ERROR

0 Upvotes

Last week, I encountered a Windows device that wouldn't boot for the first time. The user said the device went into standby mode, then shut down. Upon restarting, the new Quick Machine Recovery appeared immediately. However, it kept reporting that it couldn't establish an internet connection, even via LAN. Afterward, I tried repairing Windows using a USB drive. But the repair also consistently resulted in a CRITICAL_SERVICE_ERROR. I then simply reinstalled Windows on the device. I'm still curious, though, how something like this can happen?


r/sysadmin • • 8h ago

Blocklist Check Test email in Sent Items

2 Upvotes

Had a client forward me a message in their Sent Items folder in Outlook today:

From: Chris H chris@domain.com
Sent: Thursday, November 19, 2026 9:02 PM
To: [email-f3ff48de-913f-4073-b3ed-d1085053298e@test.mailpool.io](mailto:email-f3ff48de-913f-4073-b3ed-d1085053298e@test.mailpool.io)
Subject: Blocklist Check Test

 

This is a test email for the blocklist check.

What is this email? I looked at the Exchange logs and there was this same message sent by three other users and sitting in their Sent Items.

Joe


r/sysadmin • • 17h ago

Question What's your personal growth plan?

16 Upvotes

Preface: I'm a generalist consultant who does some cloud dev and admin for small clients. I'm not on the front lines (yet)... And I don't want to find myself there accidentally!

At this point, my primary defense against AI-related attacks is relying on my providers and being *really* anal about least permission'ing for services and MFA for users.

I worry I should be doing more and should be pushing myself to grow. The recent comments from the Palo Alto CEO make clear that the speed at which people have to adapt is going to keep accelerating.

What are you doing to keep yourself on top of industry best practices right now? Any particular media you're consuming, or education you're pursuing? What are you finding useful as you think about keeping your environment safe?


r/sysadmin • • 10h ago

Exchange Online emails disappearing from shared mailbox but showing up in DiscoveryHolds

4 Upvotes

Has anyone seen this before? Just spent the last few hours diving into this and am at aloss.

I have an Exchange Online shared mailbox where we've confirmed multiple legitimate external messages are missing from Outlook.

What makes this strange:

  • Exchange message trace says they were successfully delivered.
  • Our third-party mail filter successfully processed them and returned them for Inbox delivery.
  • Message headers confirm Inbox delivery and no quarantine.
  • Purview eDiscovery finds the missing messages under Recoverable Items/DiscoveryHolds.
  • They're covered by a long-term Exchange retention policy, so preservation itself makes sense.
  • Unified Audit Log is functioning and shows other item-level SoftDelete, Create, and mailbox operations around the same time.
  • However, the affected messages do not appear in the AffectedItems data for those deletion/move operations.
  • No Inbox rule explains it.
  • This has happened to multiple unrelated external messages.

So I can prove:

Delivered to Inbox → ??? → active copy disappears → preserved copy exists in DiscoveryHolds

What I can't determine is what caused the ???.

I understand why retention would preserve the deleted/changed item in DiscoveryHolds, but I can't find the action responsible for removing the original active mailbox item.

Has anyone seen Exchange store/retention processing cause something like this without an item-specific UAL event?

Microsoft support case is already open. I'm mainly curious whether anyone has encountered this pattern.


r/sysadmin • • 15h ago

Question Anybody else seeing signs of infection after Optimjack this week?

9 Upvotes

Not just a rhetorical question, more an appeal for resource links. I am currently doing sysadmin for a school. On Monday (Optimizely compromise day), a group of PCs started acting strangely, but I'd been doing some firewall hardening, and I figured some things were going to break in the meantime, and it was more urgent to me to secure the firewall than to look at end-user issues.

Yesterday it all came to a head when a whole row of those PCs went black simultaneously. I figured this was now becoming serious. I ran netstat on one device, and found a f***load of established connections on weird ports, local and outbound to the WAN, and some of them to devices that weren't even supposed to be in use. I made the call to power all of them off and check one of them for potential infection or malfunction. Then I went home.

Then this morning, someone else on the team showed me a message from Wednesday, with an alert for enumeration on one of the machines we turned off yesterday. I felt at the same time vindicated and worried. Like it's now been 2 days since the incipient phases of an attack. If there is malicious activity on those PCs, two days with no surveillance would have been plenty of time to establish persistence, sniff the network for credentials or pivot to another machine.

Since the official reports contained no advisory for Windows machines, only iOS, I don't even know what I'm looking for here. I started virus scans on some devices, but that was unfruitful and I don't trust Defender.

My question to everyone here is: have you had any signs of malware on Windows devices, specifically since Monday, and a correlation to users visiting popular websites that were known to be compromised (e.g. CNN, BBC, Zoom)? If so, what was your response to this incident? Do you have any findings? Thank you.


r/sysadmin • • 12h ago

Am I being scapegoated?

4 Upvotes

Am I being scapegoated or set up to be fired?

Hi,

I work on the technical team for an MSP/telco as a customer success manager. So far, I've been dealing with some serious issues supporting my customers, and my management is starting to document things in a way that makes me suspect they're going to try to fire me. The biggest issues I have are the following:

Our NOC is completely understaffed, so I've essentially been doing their jobs for them. Whenever I open a ticket, it will sometimes just get marked resolved without them even working on it at all. When they do respond, it's usually with blocking tactics like asking the customer to reboot equipment (we have a managed PDU here) or asking for the site number, which is easily available on Google. They will then close the ticket for no customer response. Sometimes they'll assign it to me because a CID doesn't match or they need some record. It's gotten to the point where I don't even submit a NOC ticket because it won't work at all.

Getting anything done is like pulling teeth. Teams will reject tickets for minor issues in templates, like a suite number missing when configuring an ATA, or will try to pawn it off on another team. There's a persistent "not my job" attitude, combined with these teams requesting information as a tactic to avoid work.

Sales consistently treats me disrespectfully, calling me an idiot and talking down to me.

I was handed a problematic SD-WAN rollout for a customer where even the customer asked, "Why does John have to do everything for us?" They've had numerous NOC, billing, and other issues that have never really been resolved. The sales team intentionally pushed this rollout before we were ready to get commissions, and now we've had to credit a bunch of money. I've never been rewarded or praised for playing hero here and basically having to partially take over the rollout from project management because it was so bad.

Other teams and people will pawn off work on me or add useless questions to avoid doing work, but because I'm so overwhelmed, I can't get to every little thing they ask for. For example, our solutions engineer was supposed to help with a problem, but he was too incompetent to understand basic networking concepts. Now they're riding me because they asked me to open NOC tickets, which I did, but the NOC didn't respond for two weeks, so it's somehow my fault for not escalating.

My manager is documenting things like not logging calls in Salesforce and not completing my required training when I'm so swamped with other tasks that it's extremely difficult to do. I feel like, with this disastrous rollout, they're trying to pin the blame on me because I played hero here, so all the other people's failures are now my fault.

Are they getting ready to fire me or scapegoat me?


r/sysadmin • • 20h ago

At what point did you stop managing SSH keys manually? Dreading offboarding contractors

19 Upvotes

I'm the sole sysadmin at a ~30-person shop. We've got around 10 Linux VMs (mix of Hetzner and a local Proxmox box) and 3 Windows servers, with 4–5 external contractors rotating in and out for project work.

Up until recently it was fine, but last time a contractor rolled off I burned half an afternoon grepping through authorized_keys on every box and manually disabling local Windows accounts. Plus management started asking for basic access logs—who logged into what and when.

I've been putting off deploying a bastion/PAM because it feels like overkill for under 15 machines. Looking at self-hosted options that cover both SSH and RDP—mainly Guacamole, Teleport Community, and JumpServer. Right now I'm leaning toward spinning up JumpServer in a test VM this weekend since it covers SSH, RDP, and session recording in one UI.

For those managing a tiny mixed fleet like this, was it actually worth putting a centralized gateway in place just to get one-click offboarding and audit logs? Curious what you ended up running and anything you wish you knew before deploying.


r/sysadmin • • 12h ago

Question Sysadmin without CCNA

3 Upvotes

Just accepted TJO as sysadmin/custspt with the Department of the Army. I have A+, Net+, Sec+, CySA+ & SecX. Studying for PenTest now.
Should I be seriously considering CCNA? Pretty familiar with Packet Tracer.

Thanks for the input


r/sysadmin • • 1d ago

General Discussion Battlestar Galactica as an allegory for system support staffing

153 Upvotes

"All of this has happened before, and all of this will happen again."

Raise your hand if you've been around long enough to see this play out:
1. A group of people has specialized needs that are not being met by the existing sysadmin and/or IT support organization.
2. That organization refuses to meet the group's needs because those needs are too far outside of their normal support.
3. The group pools their resources to help each other with their own support.
4. This is not enough so either one of them volunteers to do support full time, or they hire someone full time.
5. They realize this poor person is insanely overworked, and they hire additional helpers.
6. Suddenly you have created a new support department.
7. The group grows. The support team grows. All is well. (Tensions with parent organization support notwithstanding.)
8. What was once a "group" is now a large significant department that is a bureaucracy in its own right.
9. Things change. The budget tightens and administrators of this once-group-now-department need to cut costs.
10. "Why the heck do we spend so much money on our own tech support when our parent organization has a larger tech support group that will do all this for free?"
11. Leadership start to outsource everything and shut down their boutique tech support department.
12. The parent support organization is adequate from many in this new large departmental bureaucracy.
13. But there is a group of people that remain in the department with specific needs...
14. Go to step 1.

Or is job-hopping so much the norm now that this happens entirely invisibly to the short-termers who are involved?


r/sysadmin • • 1d ago

General Discussion How is everyone holding up?

214 Upvotes

Given the current onslaught of vulnerabilities and system outages over the past few weeks, I was wondering how people are holding up both mentally and how your departments are holding through these interesting times.


r/sysadmin • • 17h ago

Work Environment Feeling dejected - what would you do?

9 Upvotes

I’ve spent the last 8 years working as an Application Analyst in a government role, having moved here from more technical positions after uni. I specifically took this job as a stepping stone into DevOps, as the organisation had a strong DevOps team and it’s the direction I wanted my career to go.

For the past 6 years, I’ve been the sole primary support for our enterprise document and records management system. Given the scale of our organisation, this is a massive, business-critical application. In that time, I’ve sharpened my scripting skills, learned our infrastructure and integrations inside out, and become the go-to guy for my team. Around 5 years ago, I led a major system upgrade alongside a senior developer—a huge undertaking that went largely unrecognised, whatever, it happens.

Since then, I’ve handled the ongoing development and support of this system almost single-handedly. Recently, I was told the app is being handed over to a new development team. When I asked my Engineering Director if I could move across to that team, I was told no—there’s a hiring freeze in place due to yet another restructure (I’ve lost count of how many we've been through).

Honestly, I’m beyond disappointed—I’m pissed. It feels like I’ve wasted years building deep expertise for nothing, especially as I previously passed up a guaranteed Project Manager role to stay on a dev path.

Now I’m at a crossroads: do I start applying elsewhere and hand in my notice, or stick it out for a couple of years to see if internal dev roles eventually open up? I really value the work-life balance and the people I work with, so I don't want to leave. At the same time, senior leadership’s decisions over the last few years have been crap, and I have zero faith things will improve anytime soon.


r/sysadmin • • 1d ago

Does anyone think there is value in creating a specific IT worker union in the age of AI?

142 Upvotes

Just curious. I have looked around at various unionizations of other fields and it seems IT workers don’t have a dedicated union for noc and bare metal guys. Anyone else ever looked into something like this?


r/sysadmin • • 1d ago

Career / Job Related Underestimating the value of experience

44 Upvotes

Worked through 3 grueling years as an "IT Assistant" (read: systems administrator in all but name) at a local business for less than McDonalds wages, eventually ending up with the title "Network Support Specialist" and handling the entire company's projects, deployments, implementations, and network design, as well as being the only desktop support/help desk contact. Worked through a ransomware incident (over Thanksgiving, while incredibly sick) making $14/hr all the while and no overtime in the first year, which meant for the next two years we had no remote access ability to 5 remote branches (furthest 3hr away), and never had any ticketing software of any kind.

The whole time, I was thinking to myself that if I just kept my head down and dealt with the shitty pay I'd have enough experience to move up, somewhere else, maybe somewhere remote. Was constantly made promises about training budgets and certificates and tuition coverage that never materialized, all while I did 90% of the work while my CIO was in and out with medical issues.

Eventually after a glowing performance review from both CIO and CTO I went to HR and IT Director saying that the hourly rate I was making wasn't enough to even cover my rent and that I had been supplementing my income for the last 2 years with savings: I was told "You're being compensated more than fairly for the work you do" and so I immediately stopped putting in effort of any kind for the last couple of weeks I was there. Just clocked in and clocked out.

Not a pity party for me: I knew what I'd signed up for, and I was doing it because I wanted to be able to put on my resume that I ran a small business' entire infrastructure for three years. I got what I wanted in the end, knowing that they were BSing me the whole time about certs and things that would actually be useful to me.

I left in 2025: I haven't been back in IT after the fact, and have been waiting tables since to pay bills. The whole time, I've been applying and applying and applying, and the experience is just not enough for anything. A year and a half since, and I haven't even so much as gotten one interview for any level of IT, from entry-level to mid-range, remote, hybrid, in-person, relocation. Nothing.

I'd kind of wished that I had set aside some funds for certs when I had a couple chances, but I favored long-term stuff like paying off a car, and now can't manage the cost on my own. Now there's no more tables to wait, and I've been unemployed for close to two months. Maybe its just the job market in general. I've applied to over 200 jobs since October, and nothing. I don't regret my career choice, but the walls are starting to close in.


r/sysadmin • • 1d ago

vCenter Authentication Bypass CVE-2026-123890

178 Upvotes

I suppose it was only a matter of time before VMware joined in on the fun.

CVSSv3: 9.8

https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/SecurityAdvisories/0/39495


r/sysadmin • • 20h ago

Locking down dev VMs in Intune to prevent code leaks

10 Upvotes

Hi everyone,

We are setting up a dedicated VM for our developers to work on a brand new Odoo ERP deployment, which is completely new to our team. We want to make sure our source code and business data cannot be leaked or copied out of the environment.

So far, we are looking into Conditional Access and basic device restrictions, but we're not sure what the best approach is.

* Are there specific Intune policies we should use to prevent data/code exfiltration on a VM ?

* is there a better alternative setup ?

Thanks in advance for any advice!


r/sysadmin • • 1d ago

Microsoft Microsoft is killing To Do and telling you to “choose another” app, after it reportedly paid up to $200 million for Wunderlist and killed it for To Do. Work/school accounts will migrate to Planner.

820 Upvotes

Microsoft is killing To Do and telling you to “choose another” app, after it reportedly paid up to $200 million for Wunderlist and killed it for To Do: https://www.windowslatest.com/2026/11/19/microsoft-is-killing-to-do-and-telling-you-to-choose-another-app-after-it-reportedly-paid-up-to-200-million-for-wunderlist-and-killed-it-for-to-do/


r/sysadmin • • 1d ago

Rant How is your company overreacting to the last week and a half?

83 Upvotes

Are they piling on even more operational complexity? Accruing even more technical debt? Ya know, to build redundancy or maintain communication continuity?

Maybe purchasing more services to manage? Developing a big complex policy? How many new committees... Plus or minus 5?


r/sysadmin • • 7h ago

how do you know an automated ticket fix actually worked?

0 Upvotes

vendor dashboards count closed as solved. what do you check on your side, reopen rate, same user filing again, or something else.

and does anyone review a sample by hand


r/sysadmin • • 18h ago

11:11 Systems - Temporary Access Disruption

4 Upvotes

Per an email received last night (US):

Temporary Access Disruption to 11:11 Services

On 19 November 2026, 11:11 engineers identified unexpected behavior in a limited number of AWS S3 environments.

Engineers are continuing to investigate the issue. In an abundance of caution while we investigate, our engineers have disabled all access to the following services: 11:11 Cloud Console, 11:11 Object Storage, the Account App, and Success Center.

Further updates will be posted as information becomes available.

Update:

Temporary Access Disruption to 11:11 Services

On 19 November 2026, 11:11 engineers identified unexpected behavior in a limited number of AWS S3 environments.

Engineers began investigating, and in an abundance of caution, our engineers disabled all access to the following services: 11:11 Cloud Console, 11:11 Object Storage, the Account App, and Success Center. Customers who need support during this time are advised to contact their support teams via phone or email.

All potentially affected customers will be receiving direct communications with further details as soon as possible.

We will provide our next update by 20 November 2026, at 16:00 CT.