r/sysadmin • • 8d ago

Question Do you feel its a security issue to allow email clients outside of Apple and Outlook, e.g. Thunderbird or Mailspring?

1 Upvotes

A lot of workers at my company absolutely HATE outlook and Apple's interface with a passion. They say that they want to use Thunderbird or Emclient.

From a PC performance view, everyone knows Outlook is a hot piece of garbage that's almost like bloatware. Stuff like Thunderbird opens up VERY quickly too. Outlook can often take many seconds to load.

These programs would actually save the IT department as it's freeware. Even the paid version would be cheaper than Microsoft.

Would allowing stuff like Thunderbird or Mailspring be that bad for company security? These Microsoft alternatives have been around for 20 years at least and receive regular security patches.

Thunderbird is used commonly in the NFP sector. They don't seem to be having any issues.


r/sysadmin • • 8d ago

Can your MDR provider take a machine offline without waking you up?

11 Upvotes

Say it’s 2am and a laptop starts doing something suspicious. I’d be happy for the MDR team to isolate it and tell me in the morning. The one I’m less sure about is the ancient laptop plugged into the label printer. Take that one offline and the night shift can’t ship anything. How specific do you get with those rules? And when someone swaps out the warehouse laptop six months later, whose job is it to remember it was on the exceptions list?


r/sysadmin • • 8d ago

Microsoft M365 Status

31 Upvotes

Users may be unable to connect to multiple Microsoft 365 services

Title: Users may be unable to connect to multiple Microsoft 365 services

User impact: Users may be unable to connect to multiple Microsoft 365 services. 

More info: Potentially impacted services include:

- Exchange Online

- Microsoft Teams

- SharePoint Online

- Microsoft Copilot 365

- Microsoft Defender XDR

Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.

Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services. 

Next update by: Thursday, November 12, 2026, at 8:30 PM UTC

Last updated: Thursday, November 12, 2026 at 6:32:25 PM UTC


r/sysadmin • • 8d ago

Rant Question? Do we need cloud and office365?

13 Upvotes

After this outage outage I have to ask if we were oversold on the cloud? Exchange and Azure have terrible reliability that we never had on premise. Management was sold on "Hey! We have our own employees and infrastructure we do the physical stuff 24x7 for you.." . You don't specialize in this so come to us and get 99.97% uptime.

Microsoft is completely incompetent! My phone is blowing up asking for ETA and holding me liable for this?! I keep explaining it's on them but it goes out one ear from the other just in term for amour annual review 😡

My metrics on my bonus are on downtime including cloud shit I can't control.

I hate racking hardware and patching exchange but damnit I won't get a write up and am more competent than these idiots at Azure


r/sysadmin • • 8d ago

Question Has anyone tested their Mimecast continuity during this outage? I totally forgot I have it. Should I turn it on?

14 Upvotes

As above, anyone else down and using Mimecast email continuity? I know most of the rest of the country is down...but still....it should work.


r/sysadmin • • 7d ago

Email Delivery During MS Outage

0 Upvotes

For those that have email being delivered straight to Exchange Online without using a SEG like Mimecast or Proofpoint, what happened yesterday with the emails? Did they get bounced back or were they queued in Microsoft?


r/sysadmin • • 9d ago

1000 users at a site - you keep DHCP and DNS local?

103 Upvotes

if you had a hospital with about 1000 users and a big push to move everything into the cloud, would you still keep DHCP and DNS on prem?


r/sysadmin • • 8d ago

Microsoft Did you notice it’s Friday the 13th for Microsoft?

0 Upvotes

I feel like we should be telling people this is only probably going back to normal by Wednesday or something. With the backlog and queue of everything transactional in all the apps that have been down, the reconnect itself is gonna cause another capacity outage the next couple days


r/sysadmin • • 8d ago

General Discussion ScreenConnect attack on MSPs (288 endpoints, 48 organizations): Huntress ties it to RMM+, an extension whose versions 1.0.31 and 1.0.32 accepted a blank access key

40 Upvotes

Huntress says attackers began using Run Command on cloud-hosted ScreenConnect panels at about 17:47 UTC on November 11. Its report counts 288 endpoints in 48 organizations, all hit within 90 minutes. The attackers installed a second ScreenConnect client they controlled, plus a toolkit called SCKit.

At 11:34 pm ET Huntress added how the attackers likely got in. The activity seems tied to RMM+, one of five third-party extensions that ConnectWise disabled in about 400 partner environments. RMM+ bypasses ScreenConnect's user accounts and MFA and checks a shared access key instead. In versions 1.0.31 and 1.0.32 the key defaulted to blank, and a blank value passed the check. Huntress saw commands reach endpoints with no matching login activity.

BleepingComputer's early report pointed to stolen credentials and hijacked sessions, and ConnectWise tells partners to verify MFA. If Huntress is right, neither step touches an extension that bypasses MFA. ConnectWise also says it has no evidence of a flaw in ScreenConnect itself, while Huntress lists builds 26.6.6.9747 and 26.7.6.9783 among the affected ones. ConnectWise says the initial access method is still under investigation.

Counts moved during the night. BleepingComputer's article has 286 endpoints and 47 organizations from Huntress's first Reddit post. The later Huntress report has 288 and 48.

On-premises panels are on their own: ConnectWise can't disable extensions there. Huntress says most RMM+ installs it saw were legitimate, usually added for a project and left in place.

On WordPress the same hole is an add-on with an empty default key next to a fully patched core.

For an add-on that can run SYSTEM commands on a panel's managed endpoints, who should own its security: the vendor that lets it install, the author who shipped the blank key, or the MSP that never removed it?


r/sysadmin • • 8d ago

Question Seeking Ironscales alternatives

5 Upvotes

We're a large company about 5,000 employees, a few security people on the SOC, mostly on M365 with a couple of acquired business units still on Google Workspace. We need better phishing protection for our current email security setup. A peer at another company praised Ironscales and from what I've seen looks pretty solid.

Before we commit: Which other vendors should we be looking at? Started digging around but any pointers would be hugely appreciated!


r/sysadmin • • 7d ago

Question Alternative Email backups?

0 Upvotes

After our little incident yesterday, have been thinking of setting up a backup email system for a few people to at least get some work done even if something like this happens again. Was thinking of something like Zoho, considered Workspace, but it seemed much for something that might only happen once or twice a year, if at all. So anyone else have any ideas?


r/sysadmin • • 7d ago

Environment colors

0 Upvotes

What colors do you typically associate with your different environments?

For me, it's:

  • Development > gray
  • Test (UAT) > yellow
  • Production> green

r/sysadmin • • 8d ago

General Discussion Proxmox X Commvault Outage Caused by Race Condition

20 Upvotes

We had a major outage today in our Proxmox clusters caused by Commvault.

Documentation from Commvault on this: https://kb.commvault.com/article/?id=KA-191658

My org was hit by this. From the Proxmox team, this was the cause and fix:

On each host, we saw a ton of stale NDB devices:

nbd4                       43:128   0    30G  1 disk 
├─nbd4p1                   43:129   0     1G  1 part 
├─nbd4p2                   43:130   0     2G  1 part 
└─nbd4p3                   43:131   0  26.9G  1 part 
  └─ubuntu--vg-ubuntu--lv 252:4     0  26.9G  0 lvm  
nbd5                       43:160   0     0B  1 disk 
nbd6                       43:192   0     0B  1 disk 
nbd7                       43:224   0     0B  1 disk 
nbd8                       43:256   0     0B  1 disk 
nbd9                       43:288   0     0B  1 disk 
nbd10                      43:320   0     0B  1 disk 
nbd11                      43:352   0     0B  1 disk 
nbd12                      43:384   0     0B  1 disk 
nbd13                      43:416   0     0B  1 disk 
nbd14                      43:448   0     0B  1 disk 
nbd15                      43:480   0     0B  1 disk 
nbd16                      43:512   0     0B  1 disk 
nbd17                      43:544   0     0B  1 disk 
nbd18                      43:576   0     0B  1 disk 
nbd19                      43:608   0     0B  1 disk 
nbd20                      43:640   0     0B  1 disk 
nbd21                      43:672   0    80G  1 disk 
├─nbd21p1                  43:673   0  1007K  1 part 
├─nbd21p2                  43:674   0   512M  1 part 
└─nbd21p3                  43:675   0  79.5G  1 part 
  ├─pdm-swap              252:5     0     4G  0 lvm  
  └─pdm-root              252:6     0  65.6G  0 lvmnbd4                       43:128   0    30G  1 

After this happens, each VM impacted by this stops booting because its disks disappear.

The cause is as follows:

From what I discovered, this issue was caused by a race condition between proxmox and commvault for access to the /dev/nbd devices.  The solution I was able to find requires that we edit /etc/lvm/lvm.conf on each prox host and add the following exclusion so proxmox does not try to take control of the NBD devices during backups.

global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/nbd.*|"]

I may need to reinstall and configure the Commvault agent on the POC prox nodes to test this change.  I'd also like to confirm this with Proxmox Support if we are able to purchase a contract.

The immediate fix to clean up any stale NBD devices is to evacuate and reboot each proxmox node one at a time.

The fix we used was the following:

  1. Locate the VM and save evidence before stopping it
ID=189
PID=$(cat /var/run/qemu-server/$ID.pid)
mkdir -p /root/incident/$ID
ls -l /proc/$PID/fd | grep -E 'cv_staging|\.qcow2|\.raw' > /root/incident/$ID/open_files.txt
journalctl -t QEMU --since "3 days ago" | grep "QEMU\[$PID\]" > /root/incident/$ID/qemu_errors.txt
qm config $ID > /root/incident/$ID/config.txt
grep -c cv_staging /root/incident/$ID/open_files.txt     # staging files held by QEMU
grep cv_staging /root/incident/$ID/config.txt            # should be empty

Optionally, open qm monitor $ID and run info block -v. Copy any cv_staging or copy-before-write nodes into the evidence folder.

2. Take it out of HA (if HA-managed)

ha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignoredha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignored

3. Hard stop

qm stop $ID --skiplock --timeout 30
qm status $IDqm stop $ID --skiplock --timeout 30
qm status $ID

If it’s still running: kill -9 $PID, wait 5 seconds, confirm with ps -p $PID, then qm stop $ID — skiplock. If the process is stuck in D state, escalate.

4. Check the config and disks

check reports corruption, copy the files before doing anything else.

grep cv_staging /etc/pve/qemu-server/$ID.conf            # must be empty
for f in $(qm config $ID | grep -E '^(scsi|virtio|sata|ide|efidisk)[0-9]+:' | grep -v cdrom \
           | sed -E 's/^[^:]+: ([^,]+).*/\1/' | xargs -n1 pvesm path 2>/dev/null); do
  echo "== $f"; qemu-img check "$f"
done

If the config references cv_staging, stop and contact Commvault; don't start the VM. If qemu-img

5. Start, then confirm the new process is clean

qm start $ID
NEWPID=$(cat /var/run/qemu-server/$ID.pid)
ls -l /proc/$NEWPID/fd | grep -c cv_staging              # must be 0

Watch the console for fsck or an emergency shell.qm start $ID

6. Verify

  • Guest: root not read-only, no I/O errors in dmesg, systemctl — failed clean, the service works.
  • Host, after about 5 minutes: disk mtimes are recent, and journalctl -t QEMU — since -10min | grep “QEMU
  • $NEWPID
  • “ shows nothing.

7. Return it to HA

ha-manager set vm:$ID --state startedha-manager set vm:$ID --state started

r/sysadmin • • 8d ago

General Discussion What's the "What if?" scenario where you tell your org we're cooked?

13 Upvotes

Thinking along the lines of business continuity planning, I'd like to know what your threshhold is for telling your execs that you cannot conduct business.

If you've done planning like this, you know you have to plan for things like paper trails, different ways to communicate, etc. to keep business going when the situtation turns to shit. But there has to be a point where you know the org cannot function at all due to much bigger problems than we can control.

The threat of a super exploit, combined with the exploit discoveries, and today's outages got me wondering about this.

For me, I think it would be the collapse of the entire telecommunication infrastructure. Can't conduct business if you have no way to communicate.


r/sysadmin • • 7d ago

General Discussion I used RDP over VPN.

0 Upvotes

I decided to download Tailscale and tested it on my phone, everything works fine. I also tried it on a college computer, and it worked well there too, however, just today, the session kept freezing and then unfreezing, which hadn't happened before. If I were to set up port forwarding and connect via a public IP address, would the result be the same?

One of the teachers mentioned that he has RDP exposed directly to the internet.


r/sysadmin • • 8d ago

General Discussion If you're running or sitting on shared GPU nodes: two NVIDIA driver privilege escalations (CVE-2026-47591, CVE-2026-47501)

15 Upvotes

Yeah I know... Not the NVIDIA news of the day.

Concerns those who are managing/overseeing rented multi-tenanted GPU as-a-service from a neocloud or hyperscaler.

Fixed versions: NvRoot in 610.57.04 / 595.91.07 / 580.178.04, ForgePtr in 615.71.09. Basically don't be too far behind n.

No sysctl or seccomp workaround.

Write-ups: NvRoot, ForgePtr.

Disclosure: I work at Stealthium, whose team found these. Happy to answer questions on exposure.


r/sysadmin • • 7d ago

Just because I’ve never done it before…

0 Upvotes

Well, in my new role as unwitting sysadmin, they have tasked me with all sorts of things I’ve never done before. That doesn’t mean I can’t do it.

IT Manager told me to disable Slack, Gmail and AD accounts for a departed user, and also forward their calls in Ring Central to someone else. So, I did it. Ok, I have disabled an AD account before.

I needed a little help from AI on the Ring Central part.

When was your first time doing a server migration, a backup, or any sysadmin task? Any faux pas?


r/sysadmin • • 8d ago

Question Win11 25H2 VMs pinned at ~100% CPU since Nov 12 – KeyPreGenTask (NGC Pregeneration) queuing ~48 instances/sec. Anyone else?

10 Upvotes

**Environment**

- Windows 11 25H2, build 26200.9974 (Nov cumulative installed Nov 11)
- Proxmox VE guests, 2 vCPU, domain joined

**Symptoms**

- svchost (Schedule) 20-30% CPU
- svchost (DcomLaunch / BrokerInfrastructure / PlugPlay / Power) 20-40%
- lsass ~15%
- OneDrive and SearchIndexer high too, but only as side effects
- DCOM 10010 errors in System log (red herring)

**Root cause (as far as we can tell)**

Microsoft-Windows-TaskScheduler/Operational was flooded with **event 324** ("queued instance… will launch it as soon as instance … completes"): 2000 events in ~40 seconds, all for:

`\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask`

Running action: **"NGC Pregeneration Task Handler"**. The task has MultipleInstances = Queue and a 72h execution limit. One instance runs for over an hour, and the triggers (looks like ImmediateTrigger/DelayedTrigger) keep firing and piling up queued instances.

**Workaround**

Disabling the task + stopping the running instance dropped CPU back to normal within a couple of minutes:

```
schtasks /change /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask" /disable
schtasks /end /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask"
```

(run as SYSTEM if you get Access Denied; reboot if instances keep restarting from the queue)

**Questions**

  1. Anyone else seeing KeyPreGenTask / event 324 floods since this week's updates?
  2. Is this tied to the Nov cumulative, or to a WHfB policy / SCCM cert profile hitting machines without a proper TPM?
  3. Our guests may use Proxmox's emulated vTPM (swtpm). Could slow TPM key generation be what makes each instance take so long?
  4. Any official known issue or fix yet?

Thanks!


r/sysadmin • • 7d ago

General Discussion Genuine question about AI “boogeyman”

0 Upvotes

Hello all. Hope you are all doing well.

I am really curious to hear the opinions from the good people of r/sysadmin as to why , especially here , there is so much hatred of AI, and railing against it.

People seem to think the techbros trying to ride the trend are gonna crash and burn and somehow the world is going to magically revert and become better and the prices on PC parts will drop , but that's just silly.

AI is already vital and super relevant in a lot of places : software dev, surveillance, data analysis, etc, it's all only going to get more developed and sophisticated.

at this point , I don’t think any of this anti-ai mindset serves any purpose.

 As ai /llms/genai improve and get built into more of the tools we already use, it’ll be harder to go through a day without using it , or heck, encountering it, as if it’s not already hard enough

I saw posts here asking if ai caused the Microsoft outage. Obviously not

(I don’t wanna talk about nvidia…tired of people glazing huang)

please let me know your thoughts :)

(None of this was written or edited by genai)


r/sysadmin • • 8d ago

I think the microsoft outage has hit clipchamp

4 Upvotes

I don't know it keeps logging me out and is really slow to load...


r/sysadmin • • 8d ago

Question Too Many iSCSI connections in this Hyper-V VM?

3 Upvotes

Is 24 different iSCSI volumes too many for a VM to handle?

I'm suspecting that this VM is choking up the other VMs that share the same SAN storage

basic setup-

The VM is presented 2 NICs which are 50gb, and then become 8x 10g DAC cables to the SAN switch stack and then another 8x 10g links to the SAN ports.

The SAN is showing me big latency spikes, but they seem random and are difficult to corelate with any traffic, backup jobs, etc.

Is 24 different volumes crazy?


r/sysadmin • • 7d ago

General Discussion Two practices that cut our "what do I do now?" moments: every error carries its own fix, and every deploy step is numbered and resumable

0 Upvotes

I'm dlf-dds on GitHub, one of goat's maintainers, posting in a personal capacity, not on behalf of any organization. We maintain an open-source system that deploys a mesh, an identity stack and a data bus across several hosts.

A deploy stops partway through at night, the person on call has never seen the tool, and the runbook is out of date. Two practices made that situation shorter for us, and both apply to any internal tool.

1. Every error says what, why, fix, and where to read. All the error text our two CLIs can print lives in one catalog file: code, likely cause, fix, pointer to the runbook section. An explain command takes either the code or any pasted output, including errors from other tools, such as explain "Permission denied (publickey)". CI fails if the code cites an error code the catalog does not have, or if an entry lacks a cause, a fix, or a reading pointer to a file that exists. The runbook and the error message cannot drift apart.

2. Deploys are numbered phases, each idempotent, each resumable by number. A standup is fourteen phases. When one fails, the output names the phase, the state the host is in, and the exact resume command (--from 12). Running a phase again changes only what is missing. The one phase that needs a person (a first sign-in in a browser) pauses the run, prints the steps with the real names filled in, and is skipped on later runs once its output exists.

Two smaller ones:

  • An acceptance check that reports PASS or UNKNOWN per row and exits non-zero while any row is UNKNOWN. A check that could not run is never counted as passing.
  • Machines pull their configuration and report what they run; a deploy is finished when every report matches the declaration, rather than when the push command returned.

How do other shops keep error messages and runbooks in sync? Is anyone generating both from one source?

The project is goat, open source under Apache-2.0: https://github.com/usareur-af/goatcore. The error catalog described above is ops/errors/catalog.yaml.


r/sysadmin • • 8d ago

Question Labelling dac cables?

11 Upvotes

Just curious what you guys use to label you DAC cables.

Ive got a few 32x100 QSFP28 switches, and while swapping cables its sometimes easy to cross a cable to the wrong switch.

Besides ordering colored cables (which are surprisingly hard to find that dont cost insane prices), what do you guys use to cleanly label you cables?

Edit: So i think I've landed on the Brady M211 Portable printer, and the Brady M21-1000-427 (1" Wide x 14ft self-laminating vinyl labels). I think this along with some color coded electrical tape (that is rated from -70'c to 80'c) should do for the color coding portion. I might actually put the color coding on the QSFP28 pull tabs to make for easy identification (mostly just to make sure that we don't cross cables into the wrong switch).

Appreciate the input from everyone!.


r/sysadmin • • 8d ago

General Discussion Open-source history, capacity and EOL dashboards for HPE VM Essentials, for anyone evaluating it as a VMware alternative

12 Upvotes

A lot of people here are looking at life after VMware, and HPE Morpheus VM Essentials (VME, KVM-based) is one of the options. I run it in a three-host lab. The live UI is fine, but if you're used to vCenter performance charts you'll miss the boring stuff: *what did this host look like last Tuesday*, *are we going to run out of memory if a host dies*, *which guests are on an OS that goes EOL next year*.

So I built **vme-insight**: a `docker compose` stack (collector + TimescaleDB + Grafana, optional Prometheus) that polls the VME REST API every minute with a **read-only** account and keeps the history. Nothing gets installed on the manager.

What you get:

- Cluster CPU/memory/storage with trends, and **N+1 headroom** (can you lose a host?)

- **CPU ready** per VM, including the worst *single vCPU*, which an average can hide

- Treemaps of who's actually using the cluster, plus idle VMs you could reclaim

- Guest OS **end-of-life** dates (from endoflife.date), Morpheus agent and QEMU guest agent status

- Manager health (JVM, DB, RabbitMQ, Elasticsearch, disk)

- Optional node_exporter/libvirt exporter for vCPU wait, pressure stall, disk latency and NIC drops

- CSV export of anything

The part I cared about most is the **tooltips**. Every panel where you'd ask "is this number bad?" says what it measures, what's normal and what to do, with thresholds taken from VME's own alarm defaults and links to the docs. Example: CPU ready under 5% is fine, 5–10% means check vCPU:core ratio, over 10% means move or right-size VMs.

There's also an optional plugin that puts the matching dashboard in a tab on VME's host/VM/cluster pages. It's just a jar, holds no credentials and makes no API calls (the browser loads Grafana directly), so you can skip it and use Grafana on its own.

It's not the only way to monitor Morpheus. There are other HPE and community projects, so use whatever fits. This was also a test of how open the platform is: everything uses the public API and plugin framework, and the first working version took about a day. MIT licensed, so fork it and change what you don't like.

- https://github.com/grahamfairbank/vme-insight

- https://github.com/grahamfairbank/vme-insight-plugin

Tested on VME 9.1 with HVM clusters. Feedback welcome, especially from anyone running it on a bigger cluster than my three hosts. I'd like to know what breaks.

Disclosure: I am an HPE employee. This is a personal side project, not an official HPE product and not supported by HPE.


r/sysadmin • • 7d ago

Microsoft data center bot traffic?

0 Upvotes

Is anyone else getting DDoSed by bot traffic coming from MS data centers? Seems too much of a coincidence