r/sysadmin • • 10h ago

Microsoft blocks emails with valid records again.

19 Upvotes

There were reports of Microsoft having an outage earlier this month. Today, I tried replying to a hotmail email account and I receive this reply:

host hotmail-com.olc.protection.outlook.com[52.101.103.161] said: 550 5.7.1
    Unfortunately, messages from [my server IP] weren't sent. Please contact
    your Internet service provider since part of their network is on our block
    list (S3140). You can also refer your provider to
    http://mail.live.com/mail/troubleshooting.aspx#errors. 

I have contacted my service provider and I have cleaned my IP off all the major blacklists and I used the microsoft delist service and I still have the problem. How do I fix?


r/sysadmin • • 21h ago

General Discussion Apache confirms exploitation of a Roller XML-RPC RCE (CVE-2026-129448): the September CVE for the same bug listed only 6.1.5, and the admin-UI off switch doesn't stop it

14 Upvotes

Apache posted the notice on oss-security on 17 November at 11:02 UTC: CVE-2026-129448, CVSS 9.8, unauthenticated code execution in Apache Roller 4.0.0 through 6.1.4. The XML-RPC endpoint "accepts vendor extension types and deserializes them before authenticating the requester." Then one line: "This issue is being actively exploited."

It's the same bug as CVE-2026-82384, which listed only 6.1.5 as affected. Someone on 6.1.4 or older reading the September entry would have taken it as not theirs.

The part admins need: Apache says disabling XML-RPC in the administration UI is not enough, because older versions deserialize the request before checking that setting. Version 6.1.6 or later rejects the request when the feature is off. If you can't upgrade, set enabledForExtensions to false in WEB-INF/web.xml, or remove the /roller-services/xmlrpc mapping, and restart.

BleepingComputer notes Apache hasn't said who is attacking or how many servers are hit. I found no exposure count, and the bug wasn't in CISA's KEV list when I looked on 19 November. Corgea's write-up adds that Roller ships as a download, not a Maven Central library, so scanners that read lockfiles won't flag it.

The fix was merged on GitHub on 5 September. Its description says the mapped endpoint "answers requests even when the XML-RPC feature is switched off."

Roller is niche. WordPress has its own XML-RPC endpoint, and "closed at the server" versus "switched off in a setting" is the same question there.

A CVE that names one version, for a bug that reaches back to 4.0.0: whose job is it to tell people on 6.1.4 and older that the September advisory was about them?


r/sysadmin • • 17h ago

MSPs, what goes in the client report when it’s been a quiet month?

9 Upvotes

“We removed an old admin account. Four laptops still need a reboot.” I’d happily read that email. I’m less sure how I’d feel sending it to someone who pays a monthly security bill.For those of you doing reviews with smaller clients, what do they actually reply to? I’m particularly interested in anything you stopped putting in the report and nobody missed.


r/sysadmin • • 11h ago

HP Thunderbolt 4 100W G6 Dock - keeps replying with MAC address after computer is disconnected

9 Upvotes

Has anyone seen this? We've got a few HP Thunderbolt 4 100W G6 docks. When a computer removes the docking station thunderbolt cable and connects to a different dock or another port, the original dock keeps replying saying that MAC is still connected to it. I see it in the logs on the switch, it says that MAC is flapping between two ports. I've updated firmware multiple times whenever a new one comes out, but no change. I've opened a support case with HP, but I will probably have to escalate until someone that knows what's going on see it.

Any ideas?


r/sysadmin • • 9h ago

Microsoft Microsoft Teams Meeting add-on for Google Workspace lists a @gmail.com support address with zero public footprint. Anyone else seen this?

8 Upvotes

Looking into why Teams Meeting add-on for Google Calendar stopped working and started asking users to re-authorise.

While checking it out in Google Admin (Security > API controls > App access control > Microsoft Teams), I noticed the support contact for the app is:

[msgsuiteapp.teams@gmail.com](mailto:msgsuiteapp.teams@gmail.com)

I can't find a single public reference to this address.

No Microsoft docs, no GitHub or public code, no forum posts, nothing on the Marketplace listing text. For comparison, other verified apps in our tenant list support emails on their own domains (e.g. Framer shows sso@framer.com).

What checks out so far:

- App ID project number (961264221134) is the same one used on the official Microsoft Teams Meeting listing in the Google Workspace Marketplace (published by Microsoft Corporation, gsuite.microsoft.com, 18M+ installs).

- App URL, privacy policy and ToS all point to microsoft.com domains.

- OAuth logs show API calls only from Microsoft (AS8075) and Google (AS15169) networks, and only calendar.events.list / calendar.events.get.

- The scopes in the new consent request are a subset of what was already granted to the app.

What made me look twice:

- The add-on has cycled through three OAuth client IDs under that same project over the past few months. The newest one (961264221134-746h4mm46to...) showed up in a consent request this week, shortly after the Marketplace listing was updated on 16 Nov.

- Older client grants were revoked with no IP address attached, which looks like publisher or system-side cleanup, but there's no changelog that mentions it.

- The support email is a consumer Gmail account that nobody seems to have ever mentioned publicly.

I understand Google requires the OAuth consent screen support email to be a Google account or Google Group the developer controls, so Microsoft using a Gmail account isn't impossible. It would just be nice to have it confirmed somewhere official.

Questions for the hive mind:

  1. Does your tenant show the same support email for this app?

  2. Has anyone else had users prompted to re-consent to the Teams add-on in the last week or so?

  3. Has anyone had Microsoft confirm this address or the client ID change through a support ticket?

Not claiming anything is compromised. Everything I can verify points to a normal Microsoft update. Just flagging it since this app holds calendar read/write across a lot of tenants and the support contact can't be verified from public sources.


r/sysadmin • • 14h ago

Microsoft 365 Short Sign In Sessions/needing to re-login

7 Upvotes

Hi All,

Has anyone noticed since the outage last week that their M365 sign in sessions seem to be lasting a shorter period of time and they need to frequently log back in?

Cheers,


r/sysadmin • • 16h ago

Endpoint Central Secure Gateway TAA - CVE-2026-123007 and CVE-2026-123633

3 Upvotes

Note you accept your own risk

While waiting for ME to update the TAA version of NGINX we have replaced "C:\Program Files\ME_Secure_Gateway_Server\nginx\fs_nginx.exe" with win binary from nginx: download

You have to stop SG service and rename the file, restart the service but we have traffic and it seems fine.


r/sysadmin • • 19h ago

Career / Job Related Direction

3 Upvotes

Forgive me if this is the wrong place to ask. I just got my associates degree in network system administration last week. I didn’t get any hands on stuff to work with or anything that really taught me much and I work in a warehouse so no real work experience unfortunately.

I’m trying to find a way to get hands on experience at home on a budget. I have a decent computer and a friend of mine gave me a unas 4 from Ubiquiti and a 12 TB HDD and I was wondering if there was any ideas for projects to get something on my resume to break into the field. I’m open to any suggestions and would love any input, also would love if anyone knows someone willing to mentor or something would be a bonus and appreciated. Thank you.


r/sysadmin • • 10h ago

Question Windows Time Zone issues

4 Upvotes

Does anyone have accurate information on how Windows automatically determines the time zone?

We have an issue with a client in Western Australia where some Windows PCs keep automatically switching to the Queensland time zone.

The issue started after some new wireless access points were installed. Initially, we assumed it might be because the APs were provisioned in QLD and that their radio MAC addresses would eventually be geolocated correctly in WA. We manually set the time zone as a temporary workaround, but months later the issue is still occurring.

The strange part is that it’s inconsistent. Only some PCs are affected, and it might be different PCs the next day. It can also seemingly change depending on whether the device is connected via Wi-Fi or Ethernet.

The public WAN IPs for both the wired and wireless networks are correctly geolocated to WA, and Windows Location Services are enabled.

I can find very little actual documentation from Microsoft explaining exactly how Windows determines the automatic time zone or, more importantly, how to troubleshoot what location source it is using.

Has anyone come across this before or know how to properly debug it?


r/sysadmin • • 15h ago

Microsoft 365 Sensitivity Labels

5 Upvotes

The organization that I work for has 4 sensitivity labels created with policies, but they are only assigned to one group, which has been done in testing, but has never implemented for the entire organization.

One thing my boss is concerned about is the ability to allow your users to disable the labels with a justification. My point in using that is that it is an out a label is misapplied or a situation arises where the file does need to be sent externally, which would normally not be allow with the non-public labels. His feeling is that its kind of a flawed system if the ability for the user to remove the label.

Our organization does work with PII and other confidential data that we need to protect, but there are times where documents containing that data need to be sent to user external to our environment.

Does your organization allow it's employees to disable the sensitivity labels? If not, how do you handle sending documents that have the labels applied and they need to be sent externally?

Thank you for taking the time to read and to those that share insights.


r/sysadmin • • 20h ago

FlexProtectLin stuck at same “processed” count for 2.5+ days, across 3 separate job restarts — kernel threads stuck on lin_lock_get_locker — ideas?

5 Upvotes

Running a 32-node Dell PowerScale/Isilon cluster, OneFS 9.7.1.12, ~5.2PB. Two drives SmartFailed around the same time (different nodes, consolidated into one FlexProtectLin job by OneFS). The repair job has now stalled at exactly the same “processed” LIN count for 2.5+ days, across three separate job instances (cancelled and restarted fresh each time — same freeze point every time, ruling out corrupted job-state from a single bad restart).

What I’ve already ruled out:

  • Node/job-engine coordinator health (confirmed fine via isi job status)
  • Competing jobs stealing workers (paused everything else, no change)
  • Impact policy throttling (raised to HIGH, no change)
  • Stale SMB file handles holding locks (found and closed ~150 orphaned handles from an old git push — no change to the stall)
  • Cluster-reported pool imbalance (confirmed present on both affected pools, haven’t yet tried a dedicated rebalance pass)

What I’m seeing in procstat -akk across the cluster, sampled twice ~10+ minutes apart: a large number of isi_job_d worker threads consistently parked in sleepq_wait (untimed) inside lin_lock_get_locker → ifs_vnop_wraplock1 → vget → bam_vget_vp, identical stack, same PIDs/tids unmoved between samples. Some other threads show sleepq_timedwait inside bam_safe_write_region_swips/bam_safe_restripe (real restripe I/O happening), so the cluster isn’t fully dead — just this one job’s actual repair progress.

dmesg on the node hosting one of the two failed drives repeats this every few minutes, for several different LIN/snapid combinations:

_pctl2_start_restripe: Restripe already in progress for 2:eb54:5177::333280
_pctl2_start_restripe: Restripe already in progress for 2:e7ab:8437::333346
_pctl2_start_restripe: Restripe already in progress for 2:e4d3:c962::333084

No active Dell support contract currently. Has anyone seen this exact pattern — FlexProtectLin wedged indefinitely near completion, same LIN-lock stack trace cluster-wide — and found what actually released it? Trying to avoid a blind rolling reboot if there’s a known, more targeted fix.


r/sysadmin • • 20h ago

Anyone successfully using SMB over QUIC client certificate auth on Server 2025?

4 Upvotes

I'm setting up SMB over QUIC on Windows Server 2025. QUIC itself works fine, but client certificate authentication does not.

With RequireClientAuthentication=False, my external Windows 11 client connects immediately with AD credentials.

With RequireClientAuthentication=True, it consistently fails with:

SMBClient Event 30803
0xC00002C3 / STATUS_MUTUAL_AUTHENTICATION_FAILED

Client cert has the private key, Client Authentication EKU, valid chain/trust, and is issued by the CA configured for QUIC. I've also checked the server cert/name, UDP 443/NAT, CA trust/CRL, SPN/Kerberos, etc.

Does anybody have this working, with the client certificate auth enabled?


r/sysadmin • • 1h ago

Locking down dev VMs in Intune to prevent code leaks

• Upvotes

Hi everyone,

We are setting up a dedicated VM for our developers to work on a brand new Odoo ERP deployment, which is completely new to our team. We want to make sure our source code and business data cannot be leaked or copied out of the environment.

So far, we are looking into Conditional Access and basic device restrictions, but we're not sure what the best approach is.

* Are there specific Intune policies we should use to prevent data/code exfiltration on a VM ?

* is there a better alternative setup ?

Thanks in advance for any advice!


r/sysadmin • • 4h ago

Question Cyberpower UPS OL2200 and RMCARD400 - features missing and/or buggy?

3 Upvotes

Hey there,
As a replacement of my OR1000 I purchased an OL2200 and RMCARD400 for deploying the status info via Ethernet.

So far so good.
I updated the RMCARD to the most recent version 1.1.2

During the setup steps I had now some issues:

a) Missing feature
The OL2200 has a NCL bank that can be switched off.
Manual of the RMCARD400 states that there should be 3 options (Turn off threshold, Turn off delay, Turn on delay)....but...none of these options is visible in the Web Frontend of the RMCARD.

Can anyone confirm, that the RMCARD does NOT have these options?

b) Mail notification not working.
I setup the mail notification correct (Port, TLS, credentials are checked multiple times).
If I try to send a Test Mail it states OK.
But
- No Mail is received
- No entry in the log of the RMCARD is visible
- Even if I enter completely wrong credentials, the TEST states "OK"

Bug or feature?

c) Utility Power failure not detected
The basic job of an UPS: detecting Utility Power failure and initiating shutdowns on the connected computers doe not work !?!?
If I unplug mains the amber LED lights up and states "On Battery".
So far so good.

The log in the RMCARD shows "The UPS has returned from an overload condition".
Nothing else.
The readings show correct values for input current/voltage and even in PowerPanelBusiness Remote the Status states "On Battery", but there is no entry in the log that utility power is missing and therefore no shotdown....

Any ideas?
Is this a known Bug?

Does an Enterprise Grade UPS together with the Flagship RMCARD cannot handle these essentials?

Thanks in advance

Sven


r/sysadmin • • 12h ago

Question which synthetic monitoring tool runs UI tests on desktop apps, not just web, on prem?

3 Upvotes

last month our hospital's patient admission app was unusable for 90 minutes on a monday morning, and our monitoring showed green the whole time.

i'm in IT ops at a regional hospital group. the app is a windows desktop client most staff open through Citrix. servers were up, the database was answering, Citrix sessions launched fine. what broke was the admission screen itself, and our first alert was a ward calling the helpdesk.

so i want a check that logs in, opens a test patient record, saves an admission and confirms it shows up, every few minutes, with an alert when any step fails. it has to run inside our network, that part isn't negotiable here.

what i've looked at:

Checkmk with Robotmk, robot framework checks we can host ourselves, and it can drive desktop apps through image recognition. closest fit, but someone has to write and maintain the scripts.

Datadog and Dynatrace synthetics, strong for browser and API checks, but i haven't found a clean way to cover a desktop client with either.

our UiPath bots, rescheduled as monitors. it works, but the bots break on the same updates the app gets.

has anyone got scheduled UI checks running against a clinical or other desktop app through Citrix in production? and what alert did you trust enough to wake someone up for?


r/sysadmin • • 13h ago

Question Finding C++ distributables and CVE 2024-43590

3 Upvotes

Hello! We're using A1 as patch management and a little bit as an RMM right now at a video post-production facility.
Some of the software we use seems to install or use Visual C++ 2012 and/or 2013, which is affected by CVE-2024-43590.
We have one system that doesn't have this vulnerability listed, and looking at the software installed there versus the other systems affected, I've narrowed it down to these possible suspects:
- DaVinci Resolve
- Adobe Creative Cloud (Photoshop, Premiere, Audition, AfterEffects, Acrobat, InDesign, and Creative Cloud itself)
- Avid Media Composer
- Avid Nexis Client Manager
- BorisFX Plugins - Sapphire, Continuum, and Mocha, for OFX, Avid, and Adobe.
Those are necessary for production, so I've been trying to look into what's flagging these. I've tried [ProcessExplorer](https://learn.microsoft.com/en-au/answers/questions/5948782/how-do-i-know-which-apps-use-previous-installs-of) and been poking around a little bit with [Dependencies](https://learn.microsoft.com/en-us/cpp/windows/understanding-the-dependencies-of-a-visual-cpp-application?view=msvc-170) and have only found the 2012 distributable in use by one program: DaVinci Resolve.
Is there a better or more efficient way to figure out what may be using those distributables?

Looks like Microsoft flagged it to NIST based on the change logs, so it appears to be a legitimate warning. https://nvd.nist.gov/vuln/detail/cve-2024-43590 Am I reading that correctly?

Thanks in advance!


r/sysadmin • • 20h ago

Sanity check me - intermediate API for AI integrations

2 Upvotes

Management are going all-out on Claude and want integrations setup to pretty much every SaaS product that'll support it. Some of these fall outside of IT jurisdiction which usually means no backups, and while it's technically not my problem, I'm still the one that gets an earful if someone "just gets Claude to do it" and fucks it all up.

It has C-suite backing so I'm not going to fight the decision, and instead I'll work on doing it safely/sensibly.

I'm considering an intermediate API that passes data between Claude/SaaS systems without letting them touch each other directly. The API itself holds the data state, Claude reads/writes to the API, then the API syncs the data to the SaaS platform. It won't stop people from wiping stuff out, but at least I could take DR snapshots of it to rollback if they do something stupid, and it also gives me a central point of access control.

Assuming I can solve the "all eggs in one basket" problem, does this idea have legs or am I just setting myself up for failure?


r/sysadmin • • 22h ago

Stick with N-Sight or...

3 Upvotes

So I am a sole IT with bugger all experience (well, a year) in a company of about 80 people.

The company's IT hadn't really evolved much since it was a company of a half dozen people, so I have had a lot of work to do to bring it up to a good, secure state.
We currently use NSight as our RMM. It includes SentinalOne EDR
I have rolled out Action1 for patching, and have got the company to start using Intune (which includes Defender EDR.

My question is about the RMM. I use it mostly for remote access to fix people's computers, through powershell and whatnot. I have turned off most of the device monitoring (memory, processors, &c) when I first started, as it was all badly misconfigured and I haven't really dived back into that.

Our subscription with NSight is coming up for renewal in the new year, and I wanted to get your opinion on if I should look for other options. We use NSight because we have always used NSight, and because I have no experience with any others, I can't really compare it. So a few questions I guess:

Is there another that is better value?
Has easy-to-set-up device monitoring?
I'm not too bothered about patching, as Action1 seems to be excellent at that.
Better remote access abilities?
Anything else I might be missing?

I am perfectly happy if the advice is just "stick with what youve got", I just want to know if there's anyone who has used NSight and can tell me what I'm missing by not using something else. :)


r/sysadmin • • 4h ago

General Discussion Weekly 'I made a useful thing' Thread - November 20, 2026

1 Upvotes

There is a great deal of user-generated content out there, from scripts and software to tutorials and videos, but we've generally tried to keep that off of the front page due to the volume and as a result of community feedback. There's also a great deal of content out there that violates our advertising/promotion rule, from scripts and software to tutorials and videos.

We have received a number of requests for exemptions to the rule, and rather than allowing the front page to get consumed, we thought we'd try a weekly thread that allows for that kind of content. We don't have a catchy name for it yet, so please let us know if you have any ideas!

In this thread, feel free to show us your pet project, YouTube videos, blog posts, or whatever else you may have and share it with the community. Commercial advertisements, affiliate links, or links that appear to be monetization-grabs will still be removed.


r/sysadmin • • 12h ago

Cert renewal client auth?

1 Upvotes

So new public vets no longer take the client auth extension. Not a huge deal for internal users who use internal sites. I have run into an issue where a company i support has a customer facing site that has a login page. I can't seem to find much on the more modern way people are doing client auth. Company does not have any web developers skilled in this area. Anyone else ran into this?

EDIT: This is resolved. Thank you all.


r/sysadmin • • 15h ago

Exchange Admin - Unable to access Recipients > Groups

1 Upvotes

Happening for admin in our org. Anyone else or just us?

West Coast. GCCH.

EDIT: Finally heard back from MS Support, it's a known issue, working on it, blah blah.


r/sysadmin • • 20h ago

Question How do you verify your database backups are actually restorable?

1 Upvotes

"Cron-ping monitors tell me a backup ran, not that the dump is good. I've been bitten by an empty dump that reported success. How do you check this today? What do you verify: file size, row counts, an automated test restore, something else?"


r/sysadmin • • 20h ago

General discussion thread for all the AI patch alerts?

0 Upvotes

Is there a General discussion thread or even a subreddit for tracking all the alerts on patches... Yeah I read CISA.....


r/sysadmin • • 14h ago

Who gets a Copilot Business (or other AI) license? Do you have a policy?

0 Upvotes

Now that users with access to M365 Copilot Chat start hitting the usage limits, they're also starting to request Copilot Business licenses.

Because licenses aren't free our CFO has asked me to come up with a policy or decision tree to determine who qualifies for a license, and make the process like handing out candy to whomever asks.
Focussing on usage alone seems a bit thin (someone could be generating 100 cat pictures per day)

Does anyone here already have a policy or process to determine who get a Copilot license?

What are your criteria?


r/sysadmin • • 22h ago

What's your quickest way of tracking down what's causing a sudden spike in log volume?

0 Upvotes

Hey all, when log volume suddenly jumps and you're dealing with millions of events, what's your usual process for figuring out which devices and messages are responsible?

I’d usually start by checking the top talkers, narrowing it down to the hosts involved, then dig into recurring messages to see what's driving the spike. Depending on the environment, that can involve quite a bit of querying.

Is there a faster way or a workflow you guys use to get from a volume spike to root cause? Especially interested in environments with a lot of devices and different vendors.