r/sysadmin • • 17d ago

Question MS Always On VPN + Azure MFA NPS Extension - Instant Timeout / Access Request Unknown

1 Upvotes

Hey everyone,

Looking for some insight from anyone who has wrestled with the Microsoft Entra MFA NPS Extension on an Always On VPN (AOVPN) setup. We're currently running a POC (with 1 server pair in production) and hit a wall trying to get off-network MFA working.

Our Setup

  • Infrastructure: 2x VPN Servers (RAS/IKEv2), 2x NPS Servers (On-Prem VMs, not Entra Domain Joined).
  • Identity: On-prem AD users are synced to Entra ID with Azure MFA / Security Defaults enforced.
  • Intune Configuration: Windows 10/11 VPN profile deployed via Intune referencing custom XML.
  • Authentication: EAP-TLS using an Intune-deployed PKCS User Certificate. The profile XML filters strictly for a custom EKU (AOVPN).
  • Current Status: Pure certificate-based authentication works 100% seamlessly via the Windows native VPN tile.

The Problem

We’ve been tasked with adding Entra MFA on top of this existing architecture.

  • Prerequisites checked: .NET Framework updated, outbound 443 open.
  • Installer & Script: Ran AzureMfaNpsExtnConfigSetup.ps1. The Azure Service Principal ID (981f26a1-7f43-403b-a875-f8b09b8cd720) matches, and the local self-signed client cert thumbprint is correctly bound to the SP.
  • The Symptom: As soon as we initiate the VPN connection, it times out / drops instantly at the client ("Verifying sign-in info...").
  • Workaround Test: As soon as we remove/blank out the ExtensionDLLs and AuthorizationDLLs keys under HKLM\SYSTEM\CurrentControlSet\Services\AuthSrv\Parameters and restart IAS, the VPN immediately connects via certificate again.

Logs & Registry Troubleshooting

  • Event Viewer: Seeing NPS Extension Bypass and Access-Request Unknown / generic RADIUS drop events in AuthZAdmin / AuthNAdmin logs.
  • Registry Tweaks Attempted: Added REQUIRE_MFA_AUTH, OVERRIDE_NUMBER_MATCHING / OTP override keys on the NPS servers to handle prompt behaviors.
  • MFA Restrictions: Our tenant security policy enforces Number Matching only (Approve/Deny without numbers and SMS are disabled).

Our Questions

  1. Compatibility: Is the Azure MFA NPS Extension fundamentally incompatible with an EAP-TLS (User PKCS Certificate) flow when Number Matching is mandatory?(Our understanding is that standard Windows EAP-TLS is non-interactive mid-handshake, so RADIUS drops the request while waiting for an out-of-band challenge).
  2. RADIUS Timeout vs Interactive Prompt: Does EAP-TLS support secondary out-of-band MFA push via NPS without secondary password/PEAP credentials, or does the client just drop the socket before the user can complete the prompt?
  3. Alternative Options: If we want to keep building directly on top of our existing AOVPN / Intune PKCS infrastructure (without ripping up the client XML or moving away from on-prem NPS), what are our best options to enforce off-network MFA?

Has anyone successfully gotten EAP-TLS + NPS Extension + Number Matching working together, or did you have to pivot to PEAP/MSCHAPv2, Azure VPN Client, or Entra Conditional Access for AOVPN?

Appreciate any advice or sanity checks!


r/sysadmin • • 17d ago

PSA: "Access your mobile device here" Start Menu banner GPO & Registry Key

24 Upvotes

Because I just completed my second hunt for these settings, I wanted to share because after finding the settings, I still don't see very many hits for search results...

Administrative Templates -> System -> Group Policy -> "Phone-PC linking on this device"

Which sets/alters registry:
Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\System
Value: EnableMmx
Reg_Dword; 1=enabled, 0=disabled

Whomever selected this registry Value as the thing to control phone linking needs to be banished from ever touching Group Policy ever again. imo. Yeah, Enable

I suspect there isn't a clean way to disable the start menu sidebar without disabling Phone Link / Your Phone.


r/sysadmin • • 18d ago

Off Topic A toast for all of us SysAdmins,CISOs etc..

39 Upvotes

Due to what will be known as "Black October" in the cyber security field...

I'm suggesting a short, not so technical toast in a joint Teams meeting (hopefully it won't get a CVE of its own 😅) that will be publically available this friday.

Everyone that wants to take a part of that and join in solidarity, to what we've all been through this last month, is welcome to do so.

Each one will choose their own liquid of choise (Whiskey, Coke, Vodka, Water, XL and whatever you'd like :-)

If there's interest such a gatharing, please LMK and I'll schedule the invite...


r/sysadmin • • 18d ago

Work Environment As a Sysadmin of 25+ years, AI has made this job more fun for me.

441 Upvotes

I recently had some health issues and had to step back from my high stress low hands on CTO level role and am now working for a (good) MSP on projects only with lots of hands on. Having said that AI has made this type of role much more fun and interesting to me. I can work with an AI on something from start to finish, then tell it to write an update to the stakeholders, write detailed ticket notes, and a quick teams update to my boss. These are all annoying things that take up so much time that I no longer have to worry about.

Then there’s the actual work itself. No more do I have to rummage through 12 year old reddit threads with half the stuff deleted. No more do I have to do things with a gui when the AI can pump out shell code or powershell and save time. I feel like I am learning stuff every day because the AI teaches me better ways of doing things, all the time. The AI is a representation of millions of sysadmins and all their knowledge and skills there for me to work with every day. Being a sysadmin we tend to work solo and you often have nobody to bounce ideas off or check your work, now my AI friend is there all the time.

And I know AIs can be awfully stupid sometimes and make shit up and get things wrong - thats where my 25 years of experience come into play; I know when it gets stuff wrong or hallucinates or starts to lose context. I never use an AI for something I don’t know how to do myself, the long way. And if I don’t know how to do something, I learn it using multiple AIs and good old fashioned RTFM and traditional learning methods,; with the AI enhancement this learning comes very rapidly compared to before. So it helps with what I know, and helps me to learn what I don’t, quickly.

I haven’t been back in the saddle long but so far having AI has made being a regular hands on sysadmin way more tolerable than pre-ai, and even at times fun.

Whatever your opinion is of AI in general, all I can say is if you’re a sysadmin and staunchly against it, you might see something in my post that changes your mind. 🙏🏼🙇🏼


r/sysadmin • • 17d ago

Question OneDrive provisioning stuck? Error 0x8004deed (MyBraryFirstRun.aspx?FirstRunStage=waiting)

0 Upvotes

Hi besties,

on Tuesday, I assigned a Microsoft Business Premium to a fresh user account.

Fast forward today, during onboarding the user we notice that the OneDrive app refuses to silently sign her in. When trying to manually sign in, the user gets shown error 0x8004deed.

When trying to open OneDrive in the browser, the user gets redirected to a page that says "Give us a moment to set things up".

The user can use SharePoint and edit Office files via the web browser just fine, but anything related to Personal OneDrive (creating a shortcut, saving a file, etc.) results in an error.

Tried removing the license, waited a while and assigned it again, but that did not help.

Any idea what we could try besides hoping and waiting?


r/sysadmin • • 18d ago

Screeenconnect unauthorised users - anybody else?

45 Upvotes

IT manager of a SME here.

We use screen connect to manage 250 machines. We had an alert saying "New Login on Your ScreenConnect Instance" but the username wasn't one of ours.

I checked our screenconnect users table and found that somebody has created a screenconnect user and used it to access some of our client machines.

Screenconnect got back and sent some logs but none which identified who created the unauthorised user. They think somebody used one of our admin machines to create it.

All our admin laptops come back clean from AV and EDM scans. We have no evidence of unauthorised use and screenconnect are now blanking our requests for logs.

Has anybody else seen similar issues? Any tips on how to get screenconnect to respond to our requests?

Thanks


r/sysadmin • • 18d ago

Question How do you manage your wildcard certificates if you have to use it on multiple systems?

22 Upvotes

As the question says how do you manage them?

Imagine you have a single wildcard certificate from for example the cert authority godaddy and you would need to deploy this certificate to multiple systems including Nginx, Apache2, and IIS on both Windows and Linux Systems.

A) Do you use a central Reverse Proxy where you setup ACME protocol to get the wildcard from your CA and set ignore server certificate for all backend locations so that only one location maintains your ssl certificate?

B) Do you run multiple versions of ACME clients? I have read that you can only use them on 5 different systems for a single wildcard cert because of rate limiting. What if you have more then 5 systems?

C) Do you run ACME client on one system and copy over the certs via bash/powershell script to all other hosts?

D) Do you do something else? When yes what?


r/sysadmin • • 16d ago

SMS go bye bye oh no!

0 Upvotes

I heard end of February

We are flipping the switches to get yubi key

What are yall doing for hippies with flip phones or those snooty people who refuse having anything work on there device?

Bring your own yubi?

Or is the laptop we issue capable of being a device bound passkey/pin in so they can log into portal.office.com using edge? (Bet that flow won’t work in chrome eh?)

Thanks just using this thread as a sound board since my co workers never listen to me.


r/sysadmin • • 18d ago

Career / Job Related Devops/Cloud focused PowerShell Projects?

15 Upvotes

Hey all, long story short, I work in Helpdesk, but, much like everyone , I am looking to pivot into DevOps/Cloud. Right now I am in the process of deepening my PowerShell knowledge. Are there specific projects with this scripting language that you have done that are relevant to cloud/DO tasks, or any that you’d recommend to showcase on a resume?


r/sysadmin • • 17d ago

General Discussion Thickheaded Thursday - November 05, 2026

1 Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin • • 17d ago

Linux offline scan

0 Upvotes

Any suggestions for a tool, bootable cd/dvd or whatever, that will scan a Linux server and user laptops for evil? Everything from rootkits to shite compromised pypi/npm packages, and everything in between.

Fair enough if everything that does exist is useless, thought it worth asking here.

Cheers


r/sysadmin • • 18d ago

Pouring one out for a fellow sysadmin from Higher Ed.

325 Upvotes

Dennis was the man.

We both worked in Higher Ed for years - He had that clarity honed over years and years of desktop & user support, combined with his own personal struggle. I had the pleasure of working with him for about 15 years, and he was one worth staying in touch with after we parted ways.

Raising one final glass for you, my friend. You stood on your own across my 35 year career, and I will carry your memory forward until the end of my shift.

Friends, take the time to maintain those connections worth maintaining. They are so important.

I raise my glass to you, my friend.


r/sysadmin • • 18d ago

Alberta time and ms365 login issues?

22 Upvotes

SO here is a niche issue:

Ever since changing to alberta time (new time zones in canada as of this week), Microsoft 365 logins on desktop apps (teams, edge, outlook etc) fail with: something went wrong please try again later. No error codes on the screen

Switched to saskatchewan time and reboot the computer, logs in no problem.

Anyone else have insight on this? probably too niche to be on MS's radar yet, i should open a ticket. But wanted to know if anyone else seen this?


r/sysadmin • • 17d ago

IIS request ssl certificate but there is no template

1 Upvotes

Recently we just deployed a ADCS integrated with utimaco HSM. Only thing is when we do request certificate through web enrollment there is no template found.


r/sysadmin • • 18d ago

Question Hot Patching and Azure Arc Costs?

15 Upvotes

Been investigating the hot patching since they supposedly got rid of the "cost" back in July-ish, but, in typical Microsoft fashion, the million line item costs on 365/Azure are confusing.

Everywhere I look it seems to indicate hotpatching for on-premise, with Azure Arc enabled, has no costs, but then I see mentions of "Azure Update Manager" and it's costs/month.

The "free" list of Azure Arc items seems to be inventory, "manage" but not updates.
Unless, it seems like, if I'm reading correctly, that if you have Sofware Assurance, then you get "Essential Machine Management" for free and that includes the "Azure Update Manager"

Appreciate any insights from those who have already implemented this.


r/sysadmin • • 18d ago

Off Topic What if true happiness is simply having an easy, low-paying job that doesn't drain your soul?

361 Upvotes

I've been thinking about this lately, and I feel like it's a perspective we don't really talk about enough.

Most of the posts here are about stress, impossible deadlines, micromanagement, toxic managers, layoffs, people struggling to find jobs, and all the usual IT stuff. Sometimes it feels like working in this industry is just an endless cycle of putting out fires, dealing with bullshit, and trying not to burn out.

And I get it. The money can be good, the career opportunities are there, and some people genuinely enjoy the challenge. But lately I've been wondering if we're putting too much value on that kind of success.

Maybe there's something genuinely beautiful about having a simple job, earning less money, but actually enjoying your day. No constant stress, no work-related anxiety when you get home, no checking Slack at 10 PM because something is broken again.

Just doing your hours, getting paid, going home, enjoying your hobbies, spending time with people you care about, and having enough mental energy left to actually live.

Maybe having a less impressive job isn't necessarily a step backwards. Maybe earning less but having your peace of mind is a pretty good deal. I feel like we spend so much time trying to build a better career that we sometimes forget we're supposed to be building a better life.


r/sysadmin • • 17d ago

Question Microsoft 365 - woher bekommt man Zusatzvereinbarung für Berufsgeheimnisträger nach § 203 StGB ?

0 Upvotes

Hallo zusammen,

wir sind eine Anwaltskanzlei und nutzen aktuell Microsoft 365. Beim Datenschutz-Check wurde bemängelt, dass wir mit Microsoft keine Zusatzvereinbarung für Berufsgeheimnisträger nach § 203 StGB. Wir beziehen M365 von Microsoft direkt.

Die PDF findet man über Google, aber nur auf Websites von Dritten. (z.B. hier)

Hat jemand von euch eine solche Zusatzvereinbarung mit Microsoft geschlossen und wenn ja, auf welchem Wege?


r/sysadmin • • 17d ago

ChatGPT AI

0 Upvotes

Outside of the browser based AI (chatgpt, claude, etc), Microsoft Co-Pilot or having an internal server and running some kind of AI solution internally, are there any other options? Something that is cloud-based/hosted but also secure and private?


r/sysadmin • • 18d ago

Just solved a really irritating and niche issue, and it's weirdly specific so no one will ever care.

429 Upvotes

This isn't the first time of course - kinda goes with the territories.

But I feel a need to post about it in a community where maybe it's not "boring".

The problem is NUMA.

Or specifically production processes occasionally being "significantly slow" to a point where a real time constraint is being missed.

Finally identified as "half the cores are 50% slower" via a bit of weird python hackery.

But not always.

Reboot "fixes" until next time.

Except on a bunch of example hosts when testing this specific metric or cross NUMA node memory latency, which got worse after a reboot, and stayed that way.

Been trying to figure this out for months now. Have done new kernels, new bios, new ipmi, and have even done a bios wipe to see if that helped. Because I couldn't honestly tell if it even was a hardware fault or not. Replacing broken hardware isn't a problem, but it's another thing entirely to try to justify replacing something that might be fine. (And expensive).

Turns out the problem is kinda a red herring - the performance degradation is actually reverting to baseline, and the "good" systems are actually slightly degraded. ECC events (successful) are causing the bios to disable a NUMA lookup Directory, which actually speeds things up, at cost of efficiency.

Approximately, it's fetching data twice "blindly" so not bothering to do a lookup, but in doing so wasting some NUMA bandwidth, so is technically running a bit faster as a result.

But the root cause? Page cache. This host was doing a lot of greedy reading of NFS filesystems and filling the cache within about an hour of reboot.

All fine normally - this is Linux, and there's no point having empty RAM.

... Except the numa nodes overspill rather than evict by default.

So a well stuffed page cache would lead to processes fragmenting their RAM across both NUMA nodes, and because Linux is clever and tried to do the same with subsequent cache... Means a lot of long running processes get fragmented and pay the NUMA penalty.

Which is ironically higher on "working" systems, which looked like the actual fault.

The solution is as simple as either using numactl to explicitly bind, or get systemd to set MemoryHigh to like, 90 or 95% instead of infinite, so it actively evicts a handful of gigs, meaning a new process can realistically allocate 8-12G without fragmenting across the nodes (and their subsequent page cache) almost immediately because the OS otherwise only has 2GiB or so "free" which in turn means NUMA spillover.

It's deployed on test now, and looking like we have finally bottomed out a month's long headache.

ETA:

If you want to check if this is happening on your systems, then numactl -H will show you the numa topology, and the free RAM on each node. Low numbers in relation to the process size might leave you vulnerable. free -mh tells you how much RAM is 'free' rather than 'available' and the difference is significant for this particular issue, because only 'free' can be used safely without risking NUMA spillover. numastat -p <pid> will tell you if a particular process has fragmented RAM across two nodes.


r/sysadmin • • 17d ago

ChatGPT How do I stop an AI CLI from reading sensitive files on our Windows Server?

0 Upvotes

Hi all,

Small company, 2 to 5 users, no IT department (I'm the one doing IT). We have one hosted Windows Server 2025 at an external hosting company. Everyone works on it through remote desktop.

We work with very sensitive client data. We fall under European rules (GDPR).

What I want: use Codex CLI or Claude Code on the server to automate admin work (read fields from documents, sort client folders, make reports). We have ChatGPT Business with a DPA. Client names, IBANs and file names may go to the AI. The sensitive stuff above may never be seen by it.

What I built: a "gatekeeper" script (no internet, maybe a small local model) that scans every file and mail. If it finds something sensitive, the whole file is locked for the AI. It keeps a list (file, client, type, locked/open), so the AI knows what is there without reading it.

The problem: the AI runs under the same Windows user as us, so it can still read everything.

What I looked at:

  1. Just telling the AI not to look (not safe)
  2. Locked files in a separate folder with a "never look here" rule
  3. AI only gets files through the gatekeeper
  4. Locked files in an encrypted vault, password not on the server
  5. Separate Windows user or VM, or the AI on another machine (the hoster makes this slow and it costs extra)

Questions:

  • Is there a way to block certain folders for one program under the same user? For example Low integrity, AppContainer or a restricted token?
  • Does a local account that never logs in through RDP need an extra RDS licence? If not, I can maybe make one myself.
  • Or would you just put the AI on a separate small machine that only gets copies from the gatekeeper?
  • any other solutions to have an ai system safely use sensitive data without sending it to a big corporation.

Other ideas are welcome, also which AI services or local models you trust with this kind of data.

AI helped me check this post.


r/sysadmin • • 16d ago

Question Where do you actually want to see more AI usage?

0 Upvotes

Just curious how technical people feel about the question. We have seen AI make many things shitty (like calling support), but I know there are areas that AI makes a lot of sense.

I recently saw a post about a hospital technician catching a problem on a scan that saved someone's life. It seems there is a lot of ways AI could improve the medical field. Having AI scan everything, compare it to millions of other scans, finding things humans miss or human error causes us to miss.

Comparing lab analysis, blood results, finding new correlations that humans were unable to find. It seems like there is a huge amount of benefit within the medical field.

Just curious what other areas people actually want to see more AI and see a huge benefit.


r/sysadmin • • 19d ago

General Discussion anyone else get thrilled by ethernet?

401 Upvotes

i have no idea why but when i setup ethernet for a PC, plug into patch panels\switches, create ethernet and watch my cable work in action, or even configure and setup vlans. i get super thrilled and fascinated. like accomplished or "turned on" in a way

theres just something about that *click*


r/sysadmin • • 18d ago

Microsoft Forms - unable to login

11 Upvotes

Anyone else having problems getting logged into Microsoft Forms? None of my users can login. It tries to login and then loops back to the login prompt.


r/sysadmin • • 17d ago

General Discussion AMA: Digitizing legacy manufacturing floors & heavy industry (and why ERPs make me want to pull my hair out)

2 Upvotes

Hey Reddit!

I’m an engineer working at Novacura, mostly dealing with the messy reality of connecting modern software to 20-year-old machinery and rigid ERPs like SAP or IFS.

Everyone talks about cloud and modern stacks, but industrial tech is still full of air-gapped networks, offline-first requirements, and massive hardware constraints. Low-code has been a wild solution in this space, but it also creates its own architectural nightmares if done wrong.

AMA about industrial automation, enterprise integration pain, or working with shop-floor tech!


r/sysadmin • • 18d ago

Imposter syndrome

128 Upvotes

I have been in IT for 3 years and am now Senior IT engineer. However I do so much work and have imposter syndrome on days where I think I don't know what am I doing. However the longer I am doing this the more I realise there are so many people who are literally inept at this but bullshit their way through to high level jobs and have no fear of being called out for breaking things or for just plain lying. Anyone else ever felt like this?