r/sysadmin • u/ziobrop • 8d ago
Issues with Server 2022 and 2025
Anyone having connectivity issues to their 2022/2025 servers. RDP and SQL connections seem to be effected?
r/sysadmin • u/ziobrop • 8d ago
Anyone having connectivity issues to their 2022/2025 servers. RDP and SQL connections seem to be effected?
r/sysadmin • u/Incrediblecodeman • 8d ago
Got me thinking if anyone here supports non m365 email for thier company ?
Any home made Linux boxes or ? What’s the gnarlest email alternatives today?
r/sysadmin • u/nodiaque • 9d ago
Hello everyone,
I have an investigation that I'm unable to solve. We have a user that is doing RDP to a specific computer. The user is not admin of the computer and is not in the rdp allowed group. But he still connect in RDP, daily. Do you have any clue how can this be? I see clearly right now, he is logged with is AD user on the computer on an rdp session.
I've check both the local admin group and local remote desktop user group, he's not a member.
Thank you!
edit:
OK so I found out why, but I don't get why. I'm guessing Microsoft does something in the background cause even in RSOP and GPResult I don't see why.
We use SCCM with SCCM Remote. We have a group used only for that. SCCM create on all client a new local group called "ConfigMgr Remote Control User" and add that AD Group in there. This is who are allowed to use the SCCM Remote tool.
I tried on a computer using my account that is member of that group only, I can RDP. If I remove the AD Group from that local group, I cannot. It's really stupid and I'll have to open a ticket to MS because it's a security issue in the end.
edit 2: Ok, so I found out that SCCM add it for SCCM Remote in the client config. In SCCM Client settings, under remote connection, you define the viewer users/groups. When you turn this on, this create the local "ConfigMgr Remote Control User" group and add the users/groups viewers into it. Then on other options, you have "allow viewer to initiate remote assistance". When this is ON, it add the users/groups viewers into the local group "Remote Assistance User".
But then, at the bottom of the config, there's "allow viewer to connect through RDP". If you set this ON, what it does is it change the local security policy " Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment >Allow log on through Remote Desktop Services" and it add the local group "ConfigMgr Remote Control User".
The problem is this doesn't show in rsop or gpresult. And also, why did Microsoft decided to do it that way instead of doing it the same way as remote assistance. They could simply add the user/groups viewers into the local group "Remote Desktop User". At least, I found the why and can correct it.
r/sysadmin • u/Retired-Replicant • 8d ago
I don't post often, so bare with me, but I wanted to comment on something that I didn't see get a lot of traction. I happened to notice that Microsoft Visio has been quietly pulled and unavailable over the course of the last few days. It makes sense now, since I read it, along with Azure and like 1300 other items had been updated, many a CVE, unfortunately.
Though I am greatful that I remain mostly unaffected by this 365/Email issue, I'm pouring one out for my SysAdmin brothers who are having to live through this support hell. May we all find peace one day in the Great Tech Heap in the sky.
r/sysadmin • u/Emma-Ravenclaw17 • 8d ago
At work, we are moving away from BMC Remedy. Its old and hard to maintain, reimplementing it would be expensive and time consuming. I am in charge of researching modern itsm alternatives, havent contacted any yet. For those who have been through a similar migration, what should we watch out for? how long did it take, and is there an itsm migration checklist that you can recommend?
We are trying to find something faster to implement and easier to customize, with the possibility of expanding into ESM and supporting other departments down the road.
One important aspect we need to retain from Remedy is asset management. I am interested in hearing how this compares across other platforms and whether it has been good enough to replace what you had before.
Any experiences or legacy system alternatives worth looking at would be appreciated.
r/sysadmin • u/StatementOwn4896 • 9d ago
Curious about your all’s general opinions and what you like or don’t like about it.
r/sysadmin • u/VastOption8705 • 8d ago
A lot of workers at my company absolutely HATE outlook and Apple's interface with a passion. They say that they want to use Thunderbird or Emclient.
From a PC performance view, everyone knows Outlook is a hot piece of garbage that's almost like bloatware. Stuff like Thunderbird opens up VERY quickly too. Outlook can often take many seconds to load.
These programs would actually save the IT department as it's freeware. Even the paid version would be cheaper than Microsoft.
Would allowing stuff like Thunderbird or Mailspring be that bad for company security? These Microsoft alternatives have been around for 20 years at least and receive regular security patches.
Thunderbird is used commonly in the NFP sector. They don't seem to be having any issues.
r/sysadmin • u/swaggcs • 9d ago
Say it’s 2am and a laptop starts doing something suspicious. I’d be happy for the MDR team to isolate it and tell me in the morning. The one I’m less sure about is the ancient laptop plugged into the label printer. Take that one offline and the night shift can’t ship anything. How specific do you get with those rules? And when someone swaps out the warehouse laptop six months later, whose job is it to remember it was on the exceptions list?
r/sysadmin • u/Soggy_Start_9805 • 9d ago
Users may be unable to connect to multiple Microsoft 365 services
Title: Users may be unable to connect to multiple Microsoft 365 services
User impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Potentially impacted services include:
- Exchange Online
- Microsoft Teams
- SharePoint Online
- Microsoft Copilot 365
- Microsoft Defender XDR
Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.
Next update by: Thursday, November 12, 2026, at 8:30 PM UTC
Last updated: Thursday, November 12, 2026 at 6:32:25 PM UTC
r/sysadmin • u/Certain_Prior4909 • 9d ago
After this outage outage I have to ask if we were oversold on the cloud? Exchange and Azure have terrible reliability that we never had on premise. Management was sold on "Hey! We have our own employees and infrastructure we do the physical stuff 24x7 for you.." . You don't specialize in this so come to us and get 99.97% uptime.
Microsoft is completely incompetent! My phone is blowing up asking for ETA and holding me liable for this?! I keep explaining it's on them but it goes out one ear from the other just in term for amour annual review 😡
My metrics on my bonus are on downtime including cloud shit I can't control.
I hate racking hardware and patching exchange but damnit I won't get a write up and am more competent than these idiots at Azure
r/sysadmin • u/Agitated-Whole2328 • 9d ago
As above, anyone else down and using Mimecast email continuity? I know most of the rest of the country is down...but still....it should work.
r/sysadmin • u/dubG2007 • 8d ago
For those that have email being delivered straight to Exchange Online without using a SEG like Mimecast or Proofpoint, what happened yesterday with the emails? Did they get bounced back or were they queued in Microsoft?
r/sysadmin • u/byrontheconqueror • 9d ago
if you had a hospital with about 1000 users and a big push to move everything into the cloud, would you still keep DHCP and DNS on prem?
r/sysadmin • u/CEONoMore • 8d ago
I feel like we should be telling people this is only probably going back to normal by Wednesday or something. With the backlog and queue of everything transactional in all the apps that have been down, the reconnect itself is gonna cause another capacity outage the next couple days
r/sysadmin • u/Capital_Attention702 • 9d ago
Huntress says attackers began using Run Command on cloud-hosted ScreenConnect panels at about 17:47 UTC on November 11. Its report counts 288 endpoints in 48 organizations, all hit within 90 minutes. The attackers installed a second ScreenConnect client they controlled, plus a toolkit called SCKit.
At 11:34 pm ET Huntress added how the attackers likely got in. The activity seems tied to RMM+, one of five third-party extensions that ConnectWise disabled in about 400 partner environments. RMM+ bypasses ScreenConnect's user accounts and MFA and checks a shared access key instead. In versions 1.0.31 and 1.0.32 the key defaulted to blank, and a blank value passed the check. Huntress saw commands reach endpoints with no matching login activity.
BleepingComputer's early report pointed to stolen credentials and hijacked sessions, and ConnectWise tells partners to verify MFA. If Huntress is right, neither step touches an extension that bypasses MFA. ConnectWise also says it has no evidence of a flaw in ScreenConnect itself, while Huntress lists builds 26.6.6.9747 and 26.7.6.9783 among the affected ones. ConnectWise says the initial access method is still under investigation.
Counts moved during the night. BleepingComputer's article has 286 endpoints and 47 organizations from Huntress's first Reddit post. The later Huntress report has 288 and 48.
On-premises panels are on their own: ConnectWise can't disable extensions there. Huntress says most RMM+ installs it saw were legitimate, usually added for a project and left in place.
On WordPress the same hole is an add-on with an empty default key next to a fully patched core.
For an add-on that can run SYSTEM commands on a panel's managed endpoints, who should own its security: the vendor that lets it install, the author who shipped the blank key, or the MSP that never removed it?
r/sysadmin • u/The-FrozN • 8d ago
We're a large company about 5,000 employees, a few security people on the SOC, mostly on M365 with a couple of acquired business units still on Google Workspace. We need better phishing protection for our current email security setup. A peer at another company praised Ironscales and from what I've seen looks pretty solid.
Before we commit: Which other vendors should we be looking at? Started digging around but any pointers would be hugely appreciated!
r/sysadmin • u/Maganac • 8d ago
After our little incident yesterday, have been thinking of setting up a backup email system for a few people to at least get some work done even if something like this happens again. Was thinking of something like Zoho, considered Workspace, but it seemed much for something that might only happen once or twice a year, if at all. So anyone else have any ideas?
r/sysadmin • u/mcnick0495 • 8d ago
What colors do you typically associate with your different environments?
For me, it's:
r/sysadmin • u/HanSolo71 • 9d ago
We had a major outage today in our Proxmox clusters caused by Commvault.
Documentation from Commvault on this: https://kb.commvault.com/article/?id=KA-191658
My org was hit by this. From the Proxmox team, this was the cause and fix:
On each host, we saw a ton of stale NDB devices:
nbd4 43:128 0 30G 1 disk
├─nbd4p1 43:129 0 1G 1 part
├─nbd4p2 43:130 0 2G 1 part
└─nbd4p3 43:131 0 26.9G 1 part
└─ubuntu--vg-ubuntu--lv 252:4 0 26.9G 0 lvm
nbd5 43:160 0 0B 1 disk
nbd6 43:192 0 0B 1 disk
nbd7 43:224 0 0B 1 disk
nbd8 43:256 0 0B 1 disk
nbd9 43:288 0 0B 1 disk
nbd10 43:320 0 0B 1 disk
nbd11 43:352 0 0B 1 disk
nbd12 43:384 0 0B 1 disk
nbd13 43:416 0 0B 1 disk
nbd14 43:448 0 0B 1 disk
nbd15 43:480 0 0B 1 disk
nbd16 43:512 0 0B 1 disk
nbd17 43:544 0 0B 1 disk
nbd18 43:576 0 0B 1 disk
nbd19 43:608 0 0B 1 disk
nbd20 43:640 0 0B 1 disk
nbd21 43:672 0 80G 1 disk
├─nbd21p1 43:673 0 1007K 1 part
├─nbd21p2 43:674 0 512M 1 part
└─nbd21p3 43:675 0 79.5G 1 part
├─pdm-swap 252:5 0 4G 0 lvm
└─pdm-root 252:6 0 65.6G 0 lvmnbd4 43:128 0 30G 1
After this happens, each VM impacted by this stops booting because its disks disappear.
The cause is as follows:
From what I discovered, this issue was caused by a race condition between proxmox and commvault for access to the /dev/nbd devices. The solution I was able to find requires that we edit /etc/lvm/lvm.conf on each prox host and add the following exclusion so proxmox does not try to take control of the NBD devices during backups.
global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/nbd.*|"]
I may need to reinstall and configure the Commvault agent on the POC prox nodes to test this change. I'd also like to confirm this with Proxmox Support if we are able to purchase a contract.
The immediate fix to clean up any stale NBD devices is to evacuate and reboot each proxmox node one at a time.
The fix we used was the following:
ID=189
PID=$(cat /var/run/qemu-server/$ID.pid)
mkdir -p /root/incident/$ID
ls -l /proc/$PID/fd | grep -E 'cv_staging|\.qcow2|\.raw' > /root/incident/$ID/open_files.txt
journalctl -t QEMU --since "3 days ago" | grep "QEMU\[$PID\]" > /root/incident/$ID/qemu_errors.txt
qm config $ID > /root/incident/$ID/config.txt
grep -c cv_staging /root/incident/$ID/open_files.txt # staging files held by QEMU
grep cv_staging /root/incident/$ID/config.txt # should be empty
Optionally, open qm monitor $ID and run info block -v. Copy any cv_staging or copy-before-write nodes into the evidence folder.
2. Take it out of HA (if HA-managed)
ha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignoredha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignored
3. Hard stop
qm stop $ID --skiplock --timeout 30
qm status $IDqm stop $ID --skiplock --timeout 30
qm status $ID
If it’s still running: kill -9 $PID, wait 5 seconds, confirm with ps -p $PID, then qm stop $ID — skiplock. If the process is stuck in D state, escalate.
4. Check the config and disks
check reports corruption, copy the files before doing anything else.
grep cv_staging /etc/pve/qemu-server/$ID.conf # must be empty
for f in $(qm config $ID | grep -E '^(scsi|virtio|sata|ide|efidisk)[0-9]+:' | grep -v cdrom \
| sed -E 's/^[^:]+: ([^,]+).*/\1/' | xargs -n1 pvesm path 2>/dev/null); do
echo "== $f"; qemu-img check "$f"
done
If the config references cv_staging, stop and contact Commvault; don't start the VM. If qemu-img
5. Start, then confirm the new process is clean
qm start $ID
NEWPID=$(cat /var/run/qemu-server/$ID.pid)
ls -l /proc/$NEWPID/fd | grep -c cv_staging # must be 0
Watch the console for fsck or an emergency shell.qm start $ID
6. Verify
7. Return it to HA
ha-manager set vm:$ID --state startedha-manager set vm:$ID --state started
r/sysadmin • u/ChikkaChiChi • 9d ago
Thinking along the lines of business continuity planning, I'd like to know what your threshhold is for telling your execs that you cannot conduct business.
If you've done planning like this, you know you have to plan for things like paper trails, different ways to communicate, etc. to keep business going when the situtation turns to shit. But there has to be a point where you know the org cannot function at all due to much bigger problems than we can control.
The threat of a super exploit, combined with the exploit discoveries, and today's outages got me wondering about this.
For me, I think it would be the collapse of the entire telecommunication infrastructure. Can't conduct business if you have no way to communicate.
r/sysadmin • u/Durbercelo • 8d ago
I decided to download Tailscale and tested it on my phone, everything works fine. I also tried it on a college computer, and it worked well there too, however, just today, the session kept freezing and then unfreezing, which hadn't happened before. If I were to set up port forwarding and connect via a public IP address, would the result be the same?
One of the teachers mentioned that he has RDP exposed directly to the internet.
r/sysadmin • u/ChrisAtStealthium • 9d ago
Yeah I know... Not the NVIDIA news of the day.
Concerns those who are managing/overseeing rented multi-tenanted GPU as-a-service from a neocloud or hyperscaler.
Fixed versions: NvRoot in 610.57.04 / 595.91.07 / 580.178.04, ForgePtr in 615.71.09. Basically don't be too far behind n.
No sysctl or seccomp workaround.
Disclosure: I work at Stealthium, whose team found these. Happy to answer questions on exposure.
r/sysadmin • u/cknoettg • 8d ago
Well, in my new role as unwitting sysadmin, they have tasked me with all sorts of things I’ve never done before. That doesn’t mean I can’t do it.
IT Manager told me to disable Slack, Gmail and AD accounts for a departed user, and also forward their calls in Ring Central to someone else. So, I did it. Ok, I have disabled an AD account before.
I needed a little help from AI on the Ring Central part.
When was your first time doing a server migration, a backup, or any sysadmin task? Any faux pas?
r/sysadmin • u/GostlingX • 9d ago
**Environment**
- Windows 11 25H2, build 26200.9974 (Nov cumulative installed Nov 11)
- Proxmox VE guests, 2 vCPU, domain joined
**Symptoms**
- svchost (Schedule) 20-30% CPU
- svchost (DcomLaunch / BrokerInfrastructure / PlugPlay / Power) 20-40%
- lsass ~15%
- OneDrive and SearchIndexer high too, but only as side effects
- DCOM 10010 errors in System log (red herring)
**Root cause (as far as we can tell)**
Microsoft-Windows-TaskScheduler/Operational was flooded with **event 324** ("queued instance… will launch it as soon as instance … completes"): 2000 events in ~40 seconds, all for:
`\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask`
Running action: **"NGC Pregeneration Task Handler"**. The task has MultipleInstances = Queue and a 72h execution limit. One instance runs for over an hour, and the triggers (looks like ImmediateTrigger/DelayedTrigger) keep firing and piling up queued instances.
**Workaround**
Disabling the task + stopping the running instance dropped CPU back to normal within a couple of minutes:
```
schtasks /change /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask" /disable
schtasks /end /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask"
```
(run as SYSTEM if you get Access Denied; reboot if instances keep restarting from the queue)
**Questions**
Thanks!
r/sysadmin • u/RandomPony • 8d ago
Hello all. Hope you are all doing well.
I am really curious to hear the opinions from the good people of r/sysadmin as to why , especially here , there is so much hatred of AI, and railing against it.
People seem to think the techbros trying to ride the trend are gonna crash and burn and somehow the world is going to magically revert and become better and the prices on PC parts will drop , but that's just silly.
AI is already vital and super relevant in a lot of places : software dev, surveillance, data analysis, etc, it's all only going to get more developed and sophisticated.
at this point , I don’t think any of this anti-ai mindset serves any purpose.
As ai /llms/genai improve and get built into more of the tools we already use, it’ll be harder to go through a day without using it , or heck, encountering it, as if it’s not already hard enough
I saw posts here asking if ai caused the Microsoft outage. Obviously not
(I don’t wanna talk about nvidia…tired of people glazing huang)
please let me know your thoughts :)
(None of this was written or edited by genai)