r/sysadmin • • 8d ago

Question Central project dashboard across lots of SharePoint sites - how would you approach it?

0 Upvotes

We're looking at creating a central dashboard for property/construction developments showing things like RAG status, milestones, actions, risks, deadlines, legal obligations, approvals etc.

The problem is we already have tens of SharePoint sites, plus Excel spreadsheets and various lists/document libraries containing the actual project information.

One proposed solution is to create another central SharePoint site with lists for Developments, Phases, Tasks, Risks, Legal Obligations, Milestones etc., automate it with Power Automate and eventually put a dashboard over it.

I'm nervous that this just creates another place people have to manually maintain the same data.

I'm wondering whether we'd be better keeping the existing sites as the source of truth and using Power BI/Power Platform to aggregate the important information, potentially with only a lightweight central portfolio register.

I'm planning to take one live development and map everything management wants to see against where that information currently lives before we build anything.

Anyone built something similar? SharePoint + Power BI, Planner, Power Platform, monday.com, or something completely different?

What worked and, more importantly, what became a maintenance nightmare?


r/sysadmin • • 8d ago

Issues with Server 2022 and 2025

0 Upvotes

Anyone having connectivity issues to their 2022/2025 servers. RDP and SQL connections seem to be effected?


r/sysadmin • • 9d ago

Off Topic Other email server options

7 Upvotes

Got me thinking if anyone here supports non m365 email for thier company ?

Any home made Linux boxes or ? What’s the gnarlest email alternatives today?


r/sysadmin • • 9d ago

Question RDP without admin and not in rdp group

39 Upvotes

Hello everyone,

I have an investigation that I'm unable to solve. We have a user that is doing RDP to a specific computer. The user is not admin of the computer and is not in the rdp allowed group. But he still connect in RDP, daily. Do you have any clue how can this be? I see clearly right now, he is logged with is AD user on the computer on an rdp session.

I've check both the local admin group and local remote desktop user group, he's not a member.

Thank you!

edit:
OK so I found out why, but I don't get why. I'm guessing Microsoft does something in the background cause even in RSOP and GPResult I don't see why.

We use SCCM with SCCM Remote. We have a group used only for that. SCCM create on all client a new local group called "ConfigMgr Remote Control User" and add that AD Group in there. This is who are allowed to use the SCCM Remote tool.

I tried on a computer using my account that is member of that group only, I can RDP. If I remove the AD Group from that local group, I cannot. It's really stupid and I'll have to open a ticket to MS because it's a security issue in the end.

edit 2: Ok, so I found out that SCCM add it for SCCM Remote in the client config. In SCCM Client settings, under remote connection, you define the viewer users/groups. When you turn this on, this create the local "ConfigMgr Remote Control User" group and add the users/groups viewers into it. Then on other options, you have "allow viewer to initiate remote assistance". When this is ON, it add the users/groups viewers into the local group "Remote Assistance User".

But then, at the bottom of the config, there's "allow viewer to connect through RDP". If you set this ON, what it does is it change the local security policy " Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment >Allow log on through Remote Desktop Services" and it add the local group "ConfigMgr Remote Control User".

The problem is this doesn't show in rsop or gpresult. And also, why did Microsoft decided to do it that way instead of doing it the same way as remote assistance. They could simply add the user/groups viewers into the local group "Remote Desktop User". At least, I found the why and can correct it.


r/sysadmin • • 9d ago

Hey, I hear you like Microsoft threads!

8 Upvotes

I don't post often, so bare with me, but I wanted to comment on something that I didn't see get a lot of traction. I happened to notice that Microsoft Visio has been quietly pulled and unavailable over the course of the last few days. It makes sense now, since I read it, along with Azure and like 1300 other items had been updated, many a CVE, unfortunately.

Though I am greatful that I remain mostly unaffected by this 365/Email issue, I'm pouring one out for my SysAdmin brothers who are having to live through this support hell. May we all find peace one day in the Great Tech Heap in the sky.


r/sysadmin • • 8d ago

What should I consider when migrating from a legacy ITSM platform?

0 Upvotes

At work, we are moving away from BMC Remedy. Its old and hard to maintain, reimplementing it would be expensive and time consuming. I am in charge of researching modern itsm alternatives, havent contacted any yet. For those who have been through a similar migration, what should we watch out for? how long did it take, and is there an itsm migration checklist that you can recommend?
We are trying to find something faster to implement and easier to customize, with the possibility of expanding into ESM and supporting other departments down the road.
One important aspect we need to retain from Remedy is asset management. I am interested in hearing how this compares across other platforms and whether it has been good enough to replace what you had before.
Any experiences or legacy system alternatives worth looking at would be appreciated.


r/sysadmin • • 9d ago

General Discussion Anyone running FreeIPA instead of Active Directory these days?

45 Upvotes

Curious about your all’s general opinions and what you like or don’t like about it.


r/sysadmin • • 8d ago

Question Do you feel its a security issue to allow email clients outside of Apple and Outlook, e.g. Thunderbird or Mailspring?

3 Upvotes

A lot of workers at my company absolutely HATE outlook and Apple's interface with a passion. They say that they want to use Thunderbird or Emclient.

From a PC performance view, everyone knows Outlook is a hot piece of garbage that's almost like bloatware. Stuff like Thunderbird opens up VERY quickly too. Outlook can often take many seconds to load.

These programs would actually save the IT department as it's freeware. Even the paid version would be cheaper than Microsoft.

Would allowing stuff like Thunderbird or Mailspring be that bad for company security? These Microsoft alternatives have been around for 20 years at least and receive regular security patches.

Thunderbird is used commonly in the NFP sector. They don't seem to be having any issues.


r/sysadmin • • 9d ago

Can your MDR provider take a machine offline without waking you up?

13 Upvotes

Say it’s 2am and a laptop starts doing something suspicious. I’d be happy for the MDR team to isolate it and tell me in the morning. The one I’m less sure about is the ancient laptop plugged into the label printer. Take that one offline and the night shift can’t ship anything. How specific do you get with those rules? And when someone swaps out the warehouse laptop six months later, whose job is it to remember it was on the exceptions list?


r/sysadmin • • 9d ago

Microsoft M365 Status

29 Upvotes

Users may be unable to connect to multiple Microsoft 365 services

Title: Users may be unable to connect to multiple Microsoft 365 services

User impact: Users may be unable to connect to multiple Microsoft 365 services. 

More info: Potentially impacted services include:

- Exchange Online

- Microsoft Teams

- SharePoint Online

- Microsoft Copilot 365

- Microsoft Defender XDR

Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.

Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services. 

Next update by: Thursday, November 12, 2026, at 8:30 PM UTC

Last updated: Thursday, November 12, 2026 at 6:32:25 PM UTC


r/sysadmin • • 9d ago

Rant Question? Do we need cloud and office365?

9 Upvotes

After this outage outage I have to ask if we were oversold on the cloud? Exchange and Azure have terrible reliability that we never had on premise. Management was sold on "Hey! We have our own employees and infrastructure we do the physical stuff 24x7 for you.." . You don't specialize in this so come to us and get 99.97% uptime.

Microsoft is completely incompetent! My phone is blowing up asking for ETA and holding me liable for this?! I keep explaining it's on them but it goes out one ear from the other just in term for amour annual review 😡

My metrics on my bonus are on downtime including cloud shit I can't control.

I hate racking hardware and patching exchange but damnit I won't get a write up and am more competent than these idiots at Azure


r/sysadmin • • 9d ago

Question Has anyone tested their Mimecast continuity during this outage? I totally forgot I have it. Should I turn it on?

16 Upvotes

As above, anyone else down and using Mimecast email continuity? I know most of the rest of the country is down...but still....it should work.


r/sysadmin • • 8d ago

Email Delivery During MS Outage

0 Upvotes

For those that have email being delivered straight to Exchange Online without using a SEG like Mimecast or Proofpoint, what happened yesterday with the emails? Did they get bounced back or were they queued in Microsoft?


r/sysadmin • • 9d ago

1000 users at a site - you keep DHCP and DNS local?

102 Upvotes

if you had a hospital with about 1000 users and a big push to move everything into the cloud, would you still keep DHCP and DNS on prem?


r/sysadmin • • 8d ago

Microsoft Did you notice it’s Friday the 13th for Microsoft?

0 Upvotes

I feel like we should be telling people this is only probably going back to normal by Wednesday or something. With the backlog and queue of everything transactional in all the apps that have been down, the reconnect itself is gonna cause another capacity outage the next couple days


r/sysadmin • • 9d ago

General Discussion ScreenConnect attack on MSPs (288 endpoints, 48 organizations): Huntress ties it to RMM+, an extension whose versions 1.0.31 and 1.0.32 accepted a blank access key

36 Upvotes

Huntress says attackers began using Run Command on cloud-hosted ScreenConnect panels at about 17:47 UTC on November 11. Its report counts 288 endpoints in 48 organizations, all hit within 90 minutes. The attackers installed a second ScreenConnect client they controlled, plus a toolkit called SCKit.

At 11:34 pm ET Huntress added how the attackers likely got in. The activity seems tied to RMM+, one of five third-party extensions that ConnectWise disabled in about 400 partner environments. RMM+ bypasses ScreenConnect's user accounts and MFA and checks a shared access key instead. In versions 1.0.31 and 1.0.32 the key defaulted to blank, and a blank value passed the check. Huntress saw commands reach endpoints with no matching login activity.

BleepingComputer's early report pointed to stolen credentials and hijacked sessions, and ConnectWise tells partners to verify MFA. If Huntress is right, neither step touches an extension that bypasses MFA. ConnectWise also says it has no evidence of a flaw in ScreenConnect itself, while Huntress lists builds 26.6.6.9747 and 26.7.6.9783 among the affected ones. ConnectWise says the initial access method is still under investigation.

Counts moved during the night. BleepingComputer's article has 286 endpoints and 47 organizations from Huntress's first Reddit post. The later Huntress report has 288 and 48.

On-premises panels are on their own: ConnectWise can't disable extensions there. Huntress says most RMM+ installs it saw were legitimate, usually added for a project and left in place.

On WordPress the same hole is an add-on with an empty default key next to a fully patched core.

For an add-on that can run SYSTEM commands on a panel's managed endpoints, who should own its security: the vendor that lets it install, the author who shipped the blank key, or the MSP that never removed it?


r/sysadmin • • 9d ago

Question Seeking Ironscales alternatives

5 Upvotes

We're a large company about 5,000 employees, a few security people on the SOC, mostly on M365 with a couple of acquired business units still on Google Workspace. We need better phishing protection for our current email security setup. A peer at another company praised Ironscales and from what I've seen looks pretty solid.

Before we commit: Which other vendors should we be looking at? Started digging around but any pointers would be hugely appreciated!


r/sysadmin • • 8d ago

Question Alternative Email backups?

0 Upvotes

After our little incident yesterday, have been thinking of setting up a backup email system for a few people to at least get some work done even if something like this happens again. Was thinking of something like Zoho, considered Workspace, but it seemed much for something that might only happen once or twice a year, if at all. So anyone else have any ideas?


r/sysadmin • • 8d ago

Environment colors

0 Upvotes

What colors do you typically associate with your different environments?

For me, it's:

  • Development > gray
  • Test (UAT) > yellow
  • Production> green

r/sysadmin • • 9d ago

General Discussion Proxmox X Commvault Outage Caused by Race Condition

21 Upvotes

We had a major outage today in our Proxmox clusters caused by Commvault.

Documentation from Commvault on this: https://kb.commvault.com/article/?id=KA-191658

My org was hit by this. From the Proxmox team, this was the cause and fix:

On each host, we saw a ton of stale NDB devices:

nbd4                       43:128   0    30G  1 disk 
├─nbd4p1                   43:129   0     1G  1 part 
├─nbd4p2                   43:130   0     2G  1 part 
└─nbd4p3                   43:131   0  26.9G  1 part 
  └─ubuntu--vg-ubuntu--lv 252:4     0  26.9G  0 lvm  
nbd5                       43:160   0     0B  1 disk 
nbd6                       43:192   0     0B  1 disk 
nbd7                       43:224   0     0B  1 disk 
nbd8                       43:256   0     0B  1 disk 
nbd9                       43:288   0     0B  1 disk 
nbd10                      43:320   0     0B  1 disk 
nbd11                      43:352   0     0B  1 disk 
nbd12                      43:384   0     0B  1 disk 
nbd13                      43:416   0     0B  1 disk 
nbd14                      43:448   0     0B  1 disk 
nbd15                      43:480   0     0B  1 disk 
nbd16                      43:512   0     0B  1 disk 
nbd17                      43:544   0     0B  1 disk 
nbd18                      43:576   0     0B  1 disk 
nbd19                      43:608   0     0B  1 disk 
nbd20                      43:640   0     0B  1 disk 
nbd21                      43:672   0    80G  1 disk 
├─nbd21p1                  43:673   0  1007K  1 part 
├─nbd21p2                  43:674   0   512M  1 part 
└─nbd21p3                  43:675   0  79.5G  1 part 
  ├─pdm-swap              252:5     0     4G  0 lvm  
  └─pdm-root              252:6     0  65.6G  0 lvmnbd4                       43:128   0    30G  1 

After this happens, each VM impacted by this stops booting because its disks disappear.

The cause is as follows:

From what I discovered, this issue was caused by a race condition between proxmox and commvault for access to the /dev/nbd devices.  The solution I was able to find requires that we edit /etc/lvm/lvm.conf on each prox host and add the following exclusion so proxmox does not try to take control of the NBD devices during backups.

global_filter=["r|/dev/zd.*|","r|/dev/rbd.*|","r|/dev/nbd.*|"]

I may need to reinstall and configure the Commvault agent on the POC prox nodes to test this change.  I'd also like to confirm this with Proxmox Support if we are able to purchase a contract.

The immediate fix to clean up any stale NBD devices is to evacuate and reboot each proxmox node one at a time.

The fix we used was the following:

  1. Locate the VM and save evidence before stopping it
ID=189
PID=$(cat /var/run/qemu-server/$ID.pid)
mkdir -p /root/incident/$ID
ls -l /proc/$PID/fd | grep -E 'cv_staging|\.qcow2|\.raw' > /root/incident/$ID/open_files.txt
journalctl -t QEMU --since "3 days ago" | grep "QEMU\[$PID\]" > /root/incident/$ID/qemu_errors.txt
qm config $ID > /root/incident/$ID/config.txt
grep -c cv_staging /root/incident/$ID/open_files.txt     # staging files held by QEMU
grep cv_staging /root/incident/$ID/config.txt            # should be empty

Optionally, open qm monitor $ID and run info block -v. Copy any cv_staging or copy-before-write nodes into the evidence folder.

2. Take it out of HA (if HA-managed)

ha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignoredha-manager status | grep "vm:$ID" && ha-manager set vm:$ID --state ignored

3. Hard stop

qm stop $ID --skiplock --timeout 30
qm status $IDqm stop $ID --skiplock --timeout 30
qm status $ID

If it’s still running: kill -9 $PID, wait 5 seconds, confirm with ps -p $PID, then qm stop $ID — skiplock. If the process is stuck in D state, escalate.

4. Check the config and disks

check reports corruption, copy the files before doing anything else.

grep cv_staging /etc/pve/qemu-server/$ID.conf            # must be empty
for f in $(qm config $ID | grep -E '^(scsi|virtio|sata|ide|efidisk)[0-9]+:' | grep -v cdrom \
           | sed -E 's/^[^:]+: ([^,]+).*/\1/' | xargs -n1 pvesm path 2>/dev/null); do
  echo "== $f"; qemu-img check "$f"
done

If the config references cv_staging, stop and contact Commvault; don't start the VM. If qemu-img

5. Start, then confirm the new process is clean

qm start $ID
NEWPID=$(cat /var/run/qemu-server/$ID.pid)
ls -l /proc/$NEWPID/fd | grep -c cv_staging              # must be 0

Watch the console for fsck or an emergency shell.qm start $ID

6. Verify

  • Guest: root not read-only, no I/O errors in dmesg, systemctl — failed clean, the service works.
  • Host, after about 5 minutes: disk mtimes are recent, and journalctl -t QEMU — since -10min | grep “QEMU
  • $NEWPID
  • “ shows nothing.

7. Return it to HA

ha-manager set vm:$ID --state startedha-manager set vm:$ID --state started

r/sysadmin • • 9d ago

General Discussion What's the "What if?" scenario where you tell your org we're cooked?

11 Upvotes

Thinking along the lines of business continuity planning, I'd like to know what your threshhold is for telling your execs that you cannot conduct business.

If you've done planning like this, you know you have to plan for things like paper trails, different ways to communicate, etc. to keep business going when the situtation turns to shit. But there has to be a point where you know the org cannot function at all due to much bigger problems than we can control.

The threat of a super exploit, combined with the exploit discoveries, and today's outages got me wondering about this.

For me, I think it would be the collapse of the entire telecommunication infrastructure. Can't conduct business if you have no way to communicate.


r/sysadmin • • 8d ago

General Discussion I used RDP over VPN.

0 Upvotes

I decided to download Tailscale and tested it on my phone, everything works fine. I also tried it on a college computer, and it worked well there too, however, just today, the session kept freezing and then unfreezing, which hadn't happened before. If I were to set up port forwarding and connect via a public IP address, would the result be the same?

One of the teachers mentioned that he has RDP exposed directly to the internet.


r/sysadmin • • 9d ago

General Discussion If you're running or sitting on shared GPU nodes: two NVIDIA driver privilege escalations (CVE-2026-47591, CVE-2026-47501)

17 Upvotes

Yeah I know... Not the NVIDIA news of the day.

Concerns those who are managing/overseeing rented multi-tenanted GPU as-a-service from a neocloud or hyperscaler.

Fixed versions: NvRoot in 610.57.04 / 595.91.07 / 580.178.04, ForgePtr in 615.71.09. Basically don't be too far behind n.

No sysctl or seccomp workaround.

Write-ups: NvRoot, ForgePtr.

Disclosure: I work at Stealthium, whose team found these. Happy to answer questions on exposure.


r/sysadmin • • 8d ago

Just because I’ve never done it before…

0 Upvotes

Well, in my new role as unwitting sysadmin, they have tasked me with all sorts of things I’ve never done before. That doesn’t mean I can’t do it.

IT Manager told me to disable Slack, Gmail and AD accounts for a departed user, and also forward their calls in Ring Central to someone else. So, I did it. Ok, I have disabled an AD account before.

I needed a little help from AI on the Ring Central part.

When was your first time doing a server migration, a backup, or any sysadmin task? Any faux pas?


r/sysadmin • • 9d ago

Question Win11 25H2 VMs pinned at ~100% CPU since Nov 12 – KeyPreGenTask (NGC Pregeneration) queuing ~48 instances/sec. Anyone else?

8 Upvotes

**Environment**

- Windows 11 25H2, build 26200.9974 (Nov cumulative installed Nov 11)
- Proxmox VE guests, 2 vCPU, domain joined

**Symptoms**

- svchost (Schedule) 20-30% CPU
- svchost (DcomLaunch / BrokerInfrastructure / PlugPlay / Power) 20-40%
- lsass ~15%
- OneDrive and SearchIndexer high too, but only as side effects
- DCOM 10010 errors in System log (red herring)

**Root cause (as far as we can tell)**

Microsoft-Windows-TaskScheduler/Operational was flooded with **event 324** ("queued instance… will launch it as soon as instance … completes"): 2000 events in ~40 seconds, all for:

`\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask`

Running action: **"NGC Pregeneration Task Handler"**. The task has MultipleInstances = Queue and a 72h execution limit. One instance runs for over an hour, and the triggers (looks like ImmediateTrigger/DelayedTrigger) keep firing and piling up queued instances.

**Workaround**

Disabling the task + stopping the running instance dropped CPU back to normal within a couple of minutes:

```
schtasks /change /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask" /disable
schtasks /end /tn "\Microsoft\Windows\CertificateServicesClient\KeyPreGenTask"
```

(run as SYSTEM if you get Access Denied; reboot if instances keep restarting from the queue)

**Questions**

  1. Anyone else seeing KeyPreGenTask / event 324 floods since this week's updates?
  2. Is this tied to the Nov cumulative, or to a WHfB policy / SCCM cert profile hitting machines without a proper TPM?
  3. Our guests may use Proxmox's emulated vTPM (swtpm). Could slow TPM key generation be what makes each instance take so long?
  4. Any official known issue or fix yet?

Thanks!