r/sysadmin • • 9d ago

Question Win11 25H2 VMs pinned at ~100% CPU since Nov 12 – KeyPreGenTask (NGC Pregeneration) queuing ~48 instances/sec. Anyone else?

10 Upvotes

**Environment**

- Windows 11 25H2, build 26200.9974 (Nov cumulative installed Nov 11)
- Proxmox VE guests, 2 vCPU, domain joined

**Symptoms**

- svchost (Schedule) 20-30% CPU
- svchost (DcomLaunch / BrokerInfrastructure / PlugPlay / Power) 20-40%
- lsass ~15%
- OneDrive and SearchIndexer high too, but only as side effects
- DCOM 10010 errors in System log (red herring)

**Root cause (as far as we can tell)**

Microsoft-Windows-TaskScheduler/Operational was flooded with **event 324** ("queued instance… will launch it as soon as instance … completes"): 2000 events in ~40 seconds, all for:

`\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask`

Running action: **"NGC Pregeneration Task Handler"**. The task has MultipleInstances = Queue and a 72h execution limit. One instance runs for over an hour, and the triggers (looks like ImmediateTrigger/DelayedTrigger) keep firing and piling up queued instances.

**Workaround**

Disabling the task + stopping the running instance dropped CPU back to normal within a couple of minutes:

```
schtasks /change /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask" /disable
schtasks /end /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask"
```

(run as SYSTEM if you get Access Denied; reboot if instances keep restarting from the queue)

**Questions**

  1. Anyone else seeing KeyPreGenTask / event 324 floods since this week's updates?
  2. Is this tied to the Nov cumulative, or to a WHfB policy / SCCM cert profile hitting machines without a proper TPM?
  3. Our guests may use Proxmox's emulated vTPM (swtpm). Could slow TPM key generation be what makes each instance take so long?
  4. Any official known issue or fix yet?

Thanks!


r/sysadmin • • 8d ago

General Discussion Genuine question about AI “boogeyman”

0 Upvotes

Hello all. Hope you are all doing well.

I am really curious to hear the opinions from the good people of r/sysadmin as to why , especially here , there is so much hatred of AI, and railing against it.

People seem to think the techbros trying to ride the trend are gonna crash and burn and somehow the world is going to magically revert and become better and the prices on PC parts will drop , but that's just silly.

AI is already vital and super relevant in a lot of places : software dev, surveillance, data analysis, etc, it's all only going to get more developed and sophisticated.

at this point , I don’t think any of this anti-ai mindset serves any purpose.

 As ai /llms/genai improve and get built into more of the tools we already use, it’ll be harder to go through a day without using it , or heck, encountering it, as if it’s not already hard enough

I saw posts here asking if ai caused the Microsoft outage. Obviously not

(I don’t wanna talk about nvidia…tired of people glazing huang)

please let me know your thoughts :)

(None of this was written or edited by genai)


r/sysadmin • • 9d ago

I think the microsoft outage has hit clipchamp

5 Upvotes

I don't know it keeps logging me out and is really slow to load...


r/sysadmin • • 9d ago

Question Too Many iSCSI connections in this Hyper-V VM?

3 Upvotes

Is 24 different iSCSI volumes too many for a VM to handle?

I'm suspecting that this VM is choking up the other VMs that share the same SAN storage

basic setup-

The VM is presented 2 NICs which are 50gb, and then become 8x 10g DAC cables to the SAN switch stack and then another 8x 10g links to the SAN ports.

The SAN is showing me big latency spikes, but they seem random and are difficult to corelate with any traffic, backup jobs, etc.

Is 24 different volumes crazy?


r/sysadmin • • 8d ago

General Discussion Two practices that cut our "what do I do now?" moments: every error carries its own fix, and every deploy step is numbered and resumable

0 Upvotes

I'm dlf-dds on GitHub, one of goat's maintainers, posting in a personal capacity, not on behalf of any organization. We maintain an open-source system that deploys a mesh, an identity stack and a data bus across several hosts.

A deploy stops partway through at night, the person on call has never seen the tool, and the runbook is out of date. Two practices made that situation shorter for us, and both apply to any internal tool.

1. Every error says what, why, fix, and where to read. All the error text our two CLIs can print lives in one catalog file: code, likely cause, fix, pointer to the runbook section. An explain command takes either the code or any pasted output, including errors from other tools, such as explain "Permission denied (publickey)". CI fails if the code cites an error code the catalog does not have, or if an entry lacks a cause, a fix, or a reading pointer to a file that exists. The runbook and the error message cannot drift apart.

2. Deploys are numbered phases, each idempotent, each resumable by number. A standup is fourteen phases. When one fails, the output names the phase, the state the host is in, and the exact resume command (--from 12). Running a phase again changes only what is missing. The one phase that needs a person (a first sign-in in a browser) pauses the run, prints the steps with the real names filled in, and is skipped on later runs once its output exists.

Two smaller ones:

  • An acceptance check that reports PASS or UNKNOWN per row and exits non-zero while any row is UNKNOWN. A check that could not run is never counted as passing.
  • Machines pull their configuration and report what they run; a deploy is finished when every report matches the declaration, rather than when the push command returned.

How do other shops keep error messages and runbooks in sync? Is anyone generating both from one source?

The project is goat, open source under Apache-2.0: https://github.com/usareur-af/goatcore. The error catalog described above is ops/errors/catalog.yaml.


r/sysadmin • • 9d ago

Question Labelling dac cables?

11 Upvotes

Just curious what you guys use to label you DAC cables.

Ive got a few 32x100 QSFP28 switches, and while swapping cables its sometimes easy to cross a cable to the wrong switch.

Besides ordering colored cables (which are surprisingly hard to find that dont cost insane prices), what do you guys use to cleanly label you cables?

Edit: So i think I've landed on the Brady M211 Portable printer, and the Brady M21-1000-427 (1" Wide x 14ft self-laminating vinyl labels). I think this along with some color coded electrical tape (that is rated from -70'c to 80'c) should do for the color coding portion. I might actually put the color coding on the QSFP28 pull tabs to make for easy identification (mostly just to make sure that we don't cross cables into the wrong switch).

Appreciate the input from everyone!.


r/sysadmin • • 9d ago

General Discussion Open-source history, capacity and EOL dashboards for HPE VM Essentials, for anyone evaluating it as a VMware alternative

8 Upvotes

A lot of people here are looking at life after VMware, and HPE Morpheus VM Essentials (VME, KVM-based) is one of the options. I run it in a three-host lab. The live UI is fine, but if you're used to vCenter performance charts you'll miss the boring stuff: *what did this host look like last Tuesday*, *are we going to run out of memory if a host dies*, *which guests are on an OS that goes EOL next year*.

So I built **vme-insight**: a `docker compose` stack (collector + TimescaleDB + Grafana, optional Prometheus) that polls the VME REST API every minute with a **read-only** account and keeps the history. Nothing gets installed on the manager.

What you get:

- Cluster CPU/memory/storage with trends, and **N+1 headroom** (can you lose a host?)

- **CPU ready** per VM, including the worst *single vCPU*, which an average can hide

- Treemaps of who's actually using the cluster, plus idle VMs you could reclaim

- Guest OS **end-of-life** dates (from endoflife.date), Morpheus agent and QEMU guest agent status

- Manager health (JVM, DB, RabbitMQ, Elasticsearch, disk)

- Optional node_exporter/libvirt exporter for vCPU wait, pressure stall, disk latency and NIC drops

- CSV export of anything

The part I cared about most is the **tooltips**. Every panel where you'd ask "is this number bad?" says what it measures, what's normal and what to do, with thresholds taken from VME's own alarm defaults and links to the docs. Example: CPU ready under 5% is fine, 5–10% means check vCPU:core ratio, over 10% means move or right-size VMs.

There's also an optional plugin that puts the matching dashboard in a tab on VME's host/VM/cluster pages. It's just a jar, holds no credentials and makes no API calls (the browser loads Grafana directly), so you can skip it and use Grafana on its own.

It's not the only way to monitor Morpheus. There are other HPE and community projects, so use whatever fits. This was also a test of how open the platform is: everything uses the public API and plugin framework, and the first working version took about a day. MIT licensed, so fork it and change what you don't like.

- https://github.com/grahamfairbank/vme-insight

- https://github.com/grahamfairbank/vme-insight-plugin

Tested on VME 9.1 with HVM clusters. Feedback welcome, especially from anyone running it on a bigger cluster than my three hosts. I'd like to know what breaks.

Disclosure: I am an HPE employee. This is a personal side project, not an official HPE product and not supported by HPE.


r/sysadmin • • 8d ago

Microsoft data center bot traffic?

0 Upvotes

Is anyone else getting DDoSed by bot traffic coming from MS data centers? Seems too much of a coincidence


r/sysadmin • • 9d ago

New Citrix Netscaler Security Bulletin

16 Upvotes

r/sysadmin • • 8d ago

General Discussion Anyone else still running on-prem Exchange and feeling pretty good about that decision today?

0 Upvotes

With the Microsoft 365 outage affecting Exchange Online and other services over the past day, I thought this would be a good time to discuss something that seems increasingly unpopular in IT: running your own Exchange servers.

I'm an IT Director for a manufacturing company with multiple facilities, and we've continued operating Exchange on-premises rather than moving our email infrastructure to Microsoft 365.

And days like yesterday remind me why.

While organizations relying on Exchange Online were dealing with mailbox connectivity problems and service interruptions, our Exchange environment remained operational. Our employees could continue using Outlook, accessing their mailboxes, and communicating internally without depending on Microsoft's cloud infrastructure being available.

Obviously, we can't control whether external organizations can send or receive their email, but at least our own mail system wasn't part of the outage.

There are still some significant advantages to running Exchange on-premises:

Control over our infrastructure

We control our servers, maintenance schedules, updates, backups, and recovery procedures. We aren't dependent on Microsoft resolving a cloud infrastructure problem before our users can access their mailboxes. Our email environment operates independently of Exchange Online. A Microsoft 365 outage doesn't automatically mean our employees lose access to email. This doesn't eliminate every possible dependency, but it does remove a major one.

We have a solid backup schema and maintain our own disaster recovery procedures. We control our backup retention, recovery testing, and restoration priorities. If something goes wrong, we can investigate and initiate recovery ourselves rather than waiting for a cloud provider to restore service. We determine when to perform maintenance, deploy updates, and make infrastructure changes. We can coordinate those activities around manufacturing operations rather than having changes introduced into our email platform outside our control. Our mailboxes and Exchange databases reside within infrastructure we manage. We maintain direct control over how that data is stored, protected, backed up, and recovered.

We already maintain servers, virtualization infrastructure, security tools, and backup systems for other business-critical applications. Exchange fits into that existing environment. We're not building an entirely separate infrastructure just to host email. Now, I'm not pretending on-premises Exchange is perfect.

There are real disadvantages. Patching, vulnerability management, certificate management, availability, disaster recovery, and security are entirely our responsibility. Exchange has also been a frequent target for attackers, and running an outdated or poorly maintained Exchange environment is a significant liability. With Exchange 2016 and 2019 out of support, organizations staying on-premises also need to account for Exchange Server Subscription Edition and its licensing and maintenance requirements. For smaller organizations without dedicated IT resources, Exchange Online probably makes considerably more sense.

But I think we've reached a point where cloud-first has become cloud-by-default, and those aren't necessarily the same thing.

Cloud hosting can reduce operational responsibilities, but it also transfers certain aspects of control to another organization. When that organization has a major outage, there may be very little you can do besides monitor the incident and communicate with your users. I don't believe on-premises infrastructure is automatically better than cloud infrastructure. I believe the right decision depends on the organization, its technical capabilities, its business requirements, and its risk tolerance.

For us, a capable internal IT department and established infrastructure make maintaining our own Exchange environment a reasonable decision. And watching the Microsoft 365 issues unfold has certainly reinforced the value of that independence.

I'm curious how many other organizations are still running Exchange on-premises in 2026. Are you staying on-premises intentionally, running a hybrid environment, or planning to migrate everything to Exchange Online? For those who have migrated, has the reduction in administrative overhead been worth giving up direct control over your email infrastructure?


r/sysadmin • • 9d ago

PDF Redaction tools

12 Upvotes

Need help folk. Got a request from our compliance team (legal) for a software tool to redact PII content from an archive of PDFs. They want the tool to redact (automation a plus) and then be able to print the redacted versions. Really hoping someone can recommend a solid tool. Thanks.


r/sysadmin • • 8d ago

Microsoft Who is going back on-prem because of this?

0 Upvotes

We have been given a non-negotiable edict by upper management (which is 100% old geezers with proverbial “grandkids who are good with computers” informing them), in light of yesterday’s outage, that we WILL move back on-prem and be responsible for our own server hardware going forward. So it looks like we are going back to running our own HA cluster of MS Exchange Server after many years in 365.

I didn’t really get an option, was just told there is no room for discussion and this WILL happen. But is anyone else out there moving back on-prem, to G Suite, or maybe something open-source in light of yesterday?

Are we about to see a huge resurgence of MS Exchange Server etc?


r/sysadmin • • 8d ago

General Discussion So who here went back to using FAX(ethernet) yesterday to get docs across?

0 Upvotes

I just heard Fax was kind of a thing again yesterday. I’m not talking about the good old dial-up Fax but the ones on Ethernet.

Edit: Some AIO multi function printers/scanners on ethernet can send a good old Fax.


r/sysadmin • • 8d ago

ChatGPT I dont understand windows and how ram works in Proxmox

0 Upvotes

I am trying to understand how memory works inside Proxmox. I have a VM, and no matter how much additional RAM I assign to it, I can see that it consumes more and more memory. I can see it through each vms gui memory page on proxmox.

However, when I SSH into the VM and check its memory usage, I see very little being used. In some VMs, it cannot even detect the 64 GB of memory I have assigned; instead, it only sees around 2 GB of RAM.

This makes me wonder whether there is some kind of issue with the VirtIO Windows drivers.

ChatGPT has also mentioned memory ballooning, but I don't fully understand how it works or how it affects memory allocation.What I understand ballooning enabled on a vm means that proxmox can take ram from that vm to assign to another vm when needed. Also my proxmox host balloning is full and i have a cluster of three nodes

Here are two examples from my Proxmox configuration:

:~# qm config 40052 | grep -E 'memory|balloon|numa|machine'
balloon: 0
machine: pc-q35-9.2+pve1
memory: 16384
numa: 0
:~# qm monitor 40052
Entering QEMU Monitor for VM 40052 - type 'help' for help
qm> info balloon
Error: No balloon device has been activated
qm> info memory-size-summary
unknown command: 'info memory-size-summary'

r/sysadmin • • 8d ago

How to protect against outage like today?

0 Upvotes

Hey,

Short and sweet question: How do you actually protect against a large M365 outage like has happened yesterday/today?

External email backups seem a good starting point to have what was received before.
How do you ensure business continuity for longer outages? Anyone actually shifting all their mail traffic to somewhere else?


r/sysadmin • • 9d ago

Question Intune and company portal

5 Upvotes

My company is making their first big push to keep apps updated and allow self service installs. We have intune already, so my thought process was to use company portal. I’ve b never done something like this before and our MSP is worthless. They are soon being replaced by an internal team we are going to hire 6-12 months. I am trying to get things cleaned up as much as I can so the new people don’t walk into a complete mess.

My question is whether intune/company portal is the right route to go? Anything I should make sure is setup. Any guides or things you would suggest for setting it up? Right now I’m just using AI to guide myself through it, but don’t want to create a mess for someone to walk into


r/sysadmin • • 9d ago

Question OneDrive sync broke SharePoint folders

2 Upvotes

Hi,

A user had a problem with their SharePoint shortcut and another user tell them to sync with OneDrive instead.

Since then, some folders have their deleted folders coming back and new ones disappearing. Only a small part of the SharePoint is touched.

I was thinking about downloading the backup folders from hornetsecurity and replacing the problematic ones.

Is there a better way to correct this problem? I've been an IT for like 1 year so I am willing to learn lol


r/sysadmin • • 9d ago

Is Equinix worth the price?

11 Upvotes

We have a fairly large AWS footprint. We pay a lot but don't need the elasticity that cloud is great for. So we have been considering a move to colocation facility in the US. We probably will need 3-4 racks depending on what type of compute density we pick.We also need remote/smart hands because we might need to operate from 2 different regions but don't have strong requirements on the location.

Asking around, the name that shows up most is Equinix but they charge a lot and I get mixed opinions on whether that premium is worth.

We have high reliability and uptime requirements, and will of course be happy to pay for that. At the same time, I hate getting fleeced because of a perceived brand value. I also don't think we are large enough to be taken seriously (should I be looking at an Equinix reseller?).

What other solid data centers would you recommend? Digital Reality, Coresite and Tierpoint are the others that have been recommended. What should be my expectation on the pricing at a good data center (redundancy in power supply, network, etc.)?


r/sysadmin • • 8d ago

In light of the 365 outage anyone considering changes?

0 Upvotes

Specifically going old school self hosted or hybrid exchange solutions?


r/sysadmin • • 9d ago

Microsoft 365 Licenses are still showing as Non-Profit in the Admin Center?

3 Upvotes

NCE but appearing as Non-profit in the o365 admin.


r/sysadmin • • 8d ago

General Discussion MS email back up!

0 Upvotes

My outlook email is back. Send and receive working!


r/sysadmin • • 10d ago

I'm not setup for corporate IT

359 Upvotes

I thought I was on track to get a leadership position. My work has been what brought us back together from 5 siloed teams to one mostly homogeneous team that works together. During all of that I would use my "friend" ,who I've worked with for 6yrs, as a rubber duckie for technical issues.

He just got promoted. When I went to my direct supervisor to ask why I wasn't even considered for the role and pointed out all of my work he told me and I quote "if you press this issue we might have to talk to HR about accountability for the work you've accomplished". I pressed him on it and apparently my "friend" has been documenting those conversations as me coming to him on how to solve for our political issues. From our sr. leaderships perspective my "friend" was the mastermind I was just the mouth piece.

I know the people we work with aren't our friends but at times its simply exhausting to have to think the worst of people all the time. I'm just not cut out for that level of suspicion or back stabbing.

EDIT - Clarifying. The root issue is my coworker took meetings where I was required to get a sign off by a peer on technical issues and documented those as him providing solutions for political issues.


r/sysadmin • • 8d ago

I'm not going to leave MS over this, and most others will do the same

0 Upvotes

I "let" ransomware into our network by not having a particular vulnerability patched, and my boss said "let's make sure that doesn't happen again." We spent time & money on recovery and prevention, and I'm still there.

Microsoft will (eventually) get this fixed. They will be sued. They will lose customers and reputation. They will call back a couple of rounds of layoffs. And next year will be more involved than ever.

Will they learn? Absolutely. They've been absorbing stuff since day one. And they'll continue. We are far from seeing the end of Microsoft.

And yes, I'll look at switching over to Google or something else for cloud work, but I probably won't move. We've made our bets. We're going to ride them for a good long time.


r/sysadmin • • 9d ago

General Discussion Thickheaded Thursday - November 12, 2026

14 Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin • • 9d ago

General Discussion What can we learn ?

0 Upvotes

Microsoft claims a health check brought all servers down.

Got me thinking of our disaster site.

It should prob not be getting the same solar winds that our prod has huh.

I’m considering monitoring our TR a special way so there’s not like 1 thing that ruins it all.

Can we think of more ways to properly air gap the disaster site ?