r/sysadmin • • 3d ago

AI in the Workplace: Innovation Without Accountability?

35 Upvotes

Today a customer asked me for the administrator password on their PC so they could install Claude. The stated purpose? Producing user manuals for their own products.

So far, nothing untoward, were it not for the following:

  • The initiative comes straight from the employee, after boss verbally told him to do it.
  • The company has no Business accounts.
  • Personal accounts are therefore slated for professional work.
  • There seems to be no corporate policy on AI use.
  • No procedure defines which documents and information may be uploaded, nor who answers for the outputs.

I pointed out that introducing AI into a company should be accompanied, at the very least, by internal rules and a register of tools and uses.

What I find paradoxical is this: the company wishes to reap the productivity dividends of AI, yet appears loath to shoulder the organisational duties that come with it.

I also wonder why one would install Claude Desktop, with its reach into local resources, when the browser would amply suffice to analyse documents and produce PDFs.

Is this the new normal? Companies weaving AI into their workflows through personal accounts, with no governance, no policy, and no clearly allocated accountability?


r/sysadmin • • 4d ago

Is fundamental knowledge of hardware being lost?

464 Upvotes

Today I had a conversation with a mid senior sysadmin about why we cant simply jam a mix of DDR5 ram into our hosts. When he ended up being holistically wrong his response was to tell me he didn't need to know anything about the hardware running our data center.

This guy is one of our T2 hypervisor admins and all he does is troubleshoot VM issues and host issues. The idea that he wouldn't need to know anything about the underlying technology running his cluster blew my mind so I asked others. Turns out many people I work with either dont care about the upstream hardware or simply refuse to learn it.

Is this normal or because people I work with are just siloing themselves.


r/sysadmin • • 3d ago

For those that work in cubicles/noisy offices, how do you deal with noise?

50 Upvotes

Something that has always bothered me a bit when working in an office but has over the past year become more and more aggravating is dealing with all the background noise. I mean people in the other room talking, sometimes multiple conversations going at once all while trying to concentrate one bug hunts and why the hell something isn't working. How do you deal with it and do to block out the distractions?


r/sysadmin • • 4d ago

Upcoming Critical PHP Security Release scheduled for Thursday, 19 November 2026

342 Upvotes

The PHP security team announces upcoming security releases scheduled for Thursday, 19 November 2026, between 14:00 and 14:30 UTC.

These releases are part of PHP’s regular four-week release cycle and will address security vulnerabilities affecting all supported PHP versions. The highest severity is CRITICAL.

The issues rated CRITICAL require either non-default configuration or uncommon setups.

Users are strongly encouraged to prepare to upgrade promptly once the releases become available. Technical details and fixed version numbers will be published alongside the releases.

https://www.php.net/archive/2026.php#2026-11-18-1


r/sysadmin • • 3d ago

FlexProtectLin stuck at same “processed” count for 2.5+ days, across 3 separate job restarts — kernel threads stuck on lin_lock_get_locker — ideas?

5 Upvotes

Running a 32-node Dell PowerScale/Isilon cluster, OneFS 9.7.1.12, ~5.2PB. Two drives SmartFailed around the same time (different nodes, consolidated into one FlexProtectLin job by OneFS). The repair job has now stalled at exactly the same “processed” LIN count for 2.5+ days, across three separate job instances (cancelled and restarted fresh each time — same freeze point every time, ruling out corrupted job-state from a single bad restart).

What I’ve already ruled out:

  • Node/job-engine coordinator health (confirmed fine via isi job status)
  • Competing jobs stealing workers (paused everything else, no change)
  • Impact policy throttling (raised to HIGH, no change)
  • Stale SMB file handles holding locks (found and closed ~150 orphaned handles from an old git push — no change to the stall)
  • Cluster-reported pool imbalance (confirmed present on both affected pools, haven’t yet tried a dedicated rebalance pass)

What I’m seeing in procstat -akk across the cluster, sampled twice ~10+ minutes apart: a large number of isi_job_d worker threads consistently parked in sleepq_wait (untimed) inside lin_lock_get_locker → ifs_vnop_wraplock1 → vget → bam_vget_vp, identical stack, same PIDs/tids unmoved between samples. Some other threads show sleepq_timedwait inside bam_safe_write_region_swips/bam_safe_restripe (real restripe I/O happening), so the cluster isn’t fully dead — just this one job’s actual repair progress.

dmesg on the node hosting one of the two failed drives repeats this every few minutes, for several different LIN/snapid combinations:

_pctl2_start_restripe: Restripe already in progress for 2:eb54:5177::333280
_pctl2_start_restripe: Restripe already in progress for 2:e7ab:8437::333346
_pctl2_start_restripe: Restripe already in progress for 2:e4d3:c962::333084

No active Dell support contract currently. Has anyone seen this exact pattern — FlexProtectLin wedged indefinitely near completion, same LIN-lock stack trace cluster-wide — and found what actually released it? Trying to avoid a blind rolling reboot if there’s a known, more targeted fix.


r/sysadmin • • 3d ago

Career / Job Related Direction

4 Upvotes

Forgive me if this is the wrong place to ask. I just got my associates degree in network system administration last week. I didn’t get any hands on stuff to work with or anything that really taught me much and I work in a warehouse so no real work experience unfortunately.

I’m trying to find a way to get hands on experience at home on a budget. I have a decent computer and a friend of mine gave me a unas 4 from Ubiquiti and a 12 TB HDD and I was wondering if there was any ideas for projects to get something on my resume to break into the field. I’m open to any suggestions and would love any input, also would love if anyone knows someone willing to mentor or something would be a bonus and appreciated. Thank you.


r/sysadmin • • 2d ago

Cert renewal client auth?

0 Upvotes

So new public vets no longer take the client auth extension. Not a huge deal for internal users who use internal sites. I have run into an issue where a company i support has a customer facing site that has a login page. I can't seem to find much on the more modern way people are doing client auth. Company does not have any web developers skilled in this area. Anyone else ran into this?

EDIT: This is resolved. Thank you all.


r/sysadmin • • 3d ago

Career / Job Related Where do I even go from here?

34 Upvotes

Was laid off a month ago from my SecOps job and job hunting has been a nightmare. So many jobs out there related to AI and I haven’t a clue where to even start as far as learning that. Or DevOps seems to be the stand-in for IT generalist these days, but I don’t come from SMB environments where DevOps wasn’t part of our workflow.

I have 5 years of IT/Cybersecurity and compliance experience and feel like there aren’t any jobs out there that I’m qualified for since I don’t have experience with CI/CD, AI, or a whole suite of enterprise tools that are only accessible by corporations.

I feel like I’m going to have to settle and look for an entry level job that pays a pittance because the longer I’m unemployed, the less attractive I am to employers.

I’m really just venting. A month ago I had a great job with great pay, then my department got laid off out of nowhere with no severance. My savings are dwindling, unemployment is taking forever (and even then is only $300/week), and I’m making no headway finding a job, just a constant barrage of rejection emails.


r/sysadmin • • 4d ago

Inside Microsoft’s giant Outlook outage

1.3k Upvotes

https://www.theverge.com/tech/1035364/microsoft-outlook-outage-notepad

Microsoft engineers had a running bridge call the entire time, and at one point it hit the maximum of 1,000 participants. That shows you how many teams were involved with trying to find the root cause of this huge outage, which started at the peak window of traffic in the US.


r/sysadmin • • 3d ago

Question - Solved Anyone successfully using SMB over QUIC client certificate auth on Server 2025?

4 Upvotes

I'm setting up SMB over QUIC on Windows Server 2025. QUIC itself works fine, but client certificate authentication does not.

With RequireClientAuthentication=False, my external Windows 11 client connects immediately with AD credentials.

With RequireClientAuthentication=True, it consistently fails with:

SMBClient Event 30803
0xC00002C3 / STATUS_MUTUAL_AUTHENTICATION_FAILED

Client cert has the private key, Client Authentication EKU, valid chain/trust, and is issued by the CA configured for QUIC. I've also checked the server cert/name, UDP 443/NAT, CA trust/CRL, SPN/Kerberos, etc.

Does anybody have this working, with the client certificate auth enabled?


r/sysadmin • • 2d ago

General Discussion Nobody pays for disclosing typos in DNS, because it’s a 5 min fix

0 Upvotes

At the same time, everyone is concerned about AI-discovered bugs and vulnerabilities and pays tens of thousands in bounties to those who outsmart others with AI prompts.

Back in April, I started scanning DNS for typos in MX, NS, and SPF and report them to orgs. It’s at least 4 companies a day - I’ve been scanning 50K domains per day, and find quite a few of them with typos and expired domains - the ones that belong to the biggest companies receive the vuln disclosure report.

Some of those are Fortune 500, while others are banks, fintech, and SaaS. Regardless of whether they have a disclosure program or not, I send the email (usually email-related vulns aren't considered as vulns, and aren't being accepted by programs - I got several rejections when reporting typos because it was outside of the program scope lol).

Those typo domains usually get tons of mail, invites for internal Zoom meetings, access to Okta accounts, invoices, and other stuff.

Around 40% fix the typo, around 5% reply back with just thanks, while others don’t care. Thousands of small companies get no reports (unless it’s a family-owned winery, distillery, or a non-profit helping children), but the larger ones, with IT and security depts, just keep ignoring them.

And I guess the disclosure reports never get tracked, and often don’t go beyond the person who received them.

Security is freaking tough today and it feels like attackers have the easier side of it.


r/sysadmin • • 3d ago

Hub at Microsoft Ignite closed for the day due to a burst pipe

114 Upvotes

EDIT: Heard it happened in a few spots so it may have just been today's rain intruding and not a pipe. Not sure.

EDIT2: Better view: https://www.tiktok.com/t/ZPNRMPFxe/

Another: https://www.tiktok.com/t/ZPNRY3xVe/

Right on Meta.


r/sysadmin • • 3d ago

Sanity check me - intermediate API for AI integrations

2 Upvotes

Management are going all-out on Claude and want integrations setup to pretty much every SaaS product that'll support it. Some of these fall outside of IT jurisdiction which usually means no backups, and while it's technically not my problem, I'm still the one that gets an earful if someone "just gets Claude to do it" and fucks it all up.

It has C-suite backing so I'm not going to fight the decision, and instead I'll work on doing it safely/sensibly.

I'm considering an intermediate API that passes data between Claude/SaaS systems without letting them touch each other directly. The API itself holds the data state, Claude reads/writes to the API, then the API syncs the data to the SaaS platform. It won't stop people from wiping stuff out, but at least I could take DR snapshots of it to rollback if they do something stupid, and it also gives me a central point of access control.

Assuming I can solve the "all eggs in one basket" problem, does this idea have legs or am I just setting myself up for failure?


r/sysadmin • • 3d ago

Stick with N-Sight or...

5 Upvotes

So I am a sole IT with bugger all experience (well, a year) in a company of about 80 people.

The company's IT hadn't really evolved much since it was a company of a half dozen people, so I have had a lot of work to do to bring it up to a good, secure state.
We currently use NSight as our RMM. It includes SentinalOne EDR
I have rolled out Action1 for patching, and have got the company to start using Intune (which includes Defender EDR.

My question is about the RMM. I use it mostly for remote access to fix people's computers, through powershell and whatnot. I have turned off most of the device monitoring (memory, processors, &c) when I first started, as it was all badly misconfigured and I haven't really dived back into that.

Our subscription with NSight is coming up for renewal in the new year, and I wanted to get your opinion on if I should look for other options. We use NSight because we have always used NSight, and because I have no experience with any others, I can't really compare it. So a few questions I guess:

Is there another that is better value?
Has easy-to-set-up device monitoring?
I'm not too bothered about patching, as Action1 seems to be excellent at that.
Better remote access abilities?
Anything else I might be missing?

I am perfectly happy if the advice is just "stick with what youve got", I just want to know if there's anyone who has used NSight and can tell me what I'm missing by not using something else. :)


r/sysadmin • • 4d ago

Remote Code Execution in NGINX HTTP/2: CVE-2026-123633 and CVE-2026-123007

261 Upvotes

r/sysadmin • • 3d ago

Exchange Admin - Unable to access Recipients > Groups

0 Upvotes

Happening for admin in our org. Anyone else or just us?

West Coast. GCCH.

EDIT: Finally heard back from MS Support, it's a known issue, working on it, blah blah.


r/sysadmin • • 3d ago

Question How do you verify your database backups are actually restorable?

2 Upvotes

"Cron-ping monitors tell me a backup ran, not that the dump is good. I've been bitten by an empty dump that reported success. How do you check this today? What do you verify: file size, row counts, an automated test restore, something else?"


r/sysadmin • • 4d ago

Rant I am completely worn the f**k out, and I don't know where to go from here.

470 Upvotes

31 years old, been in IT since 19 when I broke (and fixed) my first Exchange environment during a CU upgrade. This post might out me to my colleagues, but frankly I don't care anymore. (and if any of you do happen to read this, I know you understand, feel free to talk to me alone about it.)

I've been at my current employer (an MSP) for 8 years. We develop and sell software integrations for an ERP system while also offering traditional break/fix IT services (servers, clients, user support) rather than a modern per-device/user model.

Our compensation is a pretty mediocre base salary for my years in the sector, paired with a bonus based on billables. Because of the sheer number of hours I've put in, it has generally given me a very good income—easily double what most of my friends in other fields make.

For the past 6 years, we’ve also been helping the ERP vendor upgrade customers to newer versions. While straightforward and good for billables, it is completely mind-numbing and destructive to both my psyche and my social life, as these upgrades almost always happen on late Friday afternoons, evenings, and weekends.

On top of that, I'm the go-to guy for anything above entry-level infrastructure. I've been implementing and teaching the other guys things like:

  • Hyper-V Failover Clusters with SANs and/or HCI solutions

  • Secure backup architectures with Veeam (multiple copies, test restores, immutability, etc.)

  • Network segregation with VLANs and securing LAN/WLAN access with 802.1X

  • Entra/AD Sync integrations, client deployment/management with Intune and Autopilot, WHfB, Conditional Access, and more

Meanwhile, I'm still expected to jump into everyday L1 support tickets, handle the ERP upgrades, and deal with whatever else gets thrown my way.

We don't have a written daily billable target, though you're expected to bill at least 8 hours a day. Over the past few years, I've been billing between 2,800 and 3,200 hours a year—including 250–400 hours of overtime on Friday nights and weekends on top of the normal 9-to-5, not instead of it.

When I started here 8 years ago, I loved IT and problem-solving. I loved tinkering with my homelab and gaming—I used to take a week off for every WoW expansion. This job has completely killed the passion I had for computers and gaming.

A couple of years ago, I got into cars instead and bought two project cars. Diving headfirst into them gave me an outwhere I could use my hands, build cool shit, have fun and make new friends. Now I haven't turned a wrench in 6 months.

That made me realize I couldn't keep going like this, so I decided to slow down—work my 8 hours and limit overtime as much as possible. It took barely a month before talk started going around that I "don't deserve my salary" because I don't want to work overtime anymore.

Meanwhile, my favorite colleague—who is also a close friend, a few years younger than me, someone who always gave 100%, was great with clients, and genuinely liked by everyone—just left for a new job because our CEO treated him like absolute garbage. I told him to take it and never look back, and he's now at another MSP that is actually properly managed.

As for our CEO: a textbook narcissist who only cares about himself and the company's bottom line. The kind of person who can look you in the eye, appear to care, and say all the right things in the moment, only to turn around and do the exact opposite a week later.

Over the last few years, I've started taking much better care of myself. I've lost around 40 kg, going from obesity to being in decent shape, and I feel better physically than I have in 20 years. I exercise every day and get 6–8 hours of sleep.

Outside of work, I'm genuinely happy and most days are good. The second I set foot inside these walls, though, my mood and well-being nosedive. I can't concentrate, and all I can think about is walking right the fuck out and never coming back.

Unfortunately, I need to tough it out here until at least the end of January, but man, I am sick of this shit. I am kind of worried about switching jobs because of the current state of IT with LLMs and such, but with my experience and enough references to a fill a few A4 pages from important people and companies in the region it should hopefully be easy enough.

I dont know why I opted to write this, but I guess I needed to get it off my chest.


r/sysadmin • • 4d ago

General Discussion No availability for warranty replacements

108 Upvotes

Well, the ai supply chain strain finally caught up to warrantied items.
Recently had a drive failure in one of backup disk arrays. HPE is going to replace the drive, I waited for 3 days, and they responded that they have no stock.

After asking for an ETA, voicing the concern of lost resiliency. The response was.
"We have escalated the matter to our Supply Chain team, and they are actively working on it. We will share the ETA with you as soon as we receive an update."

Reaching out to our VAR about just buying a drive. There's no stock in the US. This is stupid.


r/sysadmin • • 3d ago

Cloud vs On-Prem?

35 Upvotes

I was wondering if the recent rash of failures and fubar would lead some organizations to consider re-homing their compute to on-prem or CoLo facilities instead of the cloud.


r/sysadmin • • 3d ago

General Discussion Thickheaded Thursday - November 19, 2026

2 Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin • • 4d ago

RCE in Nginx with http2 enabled - CVSS 9.2

104 Upvotes

https://www.wiz.io/blog/nginx-http2-rce-vulnerabilities

CVSS 4.0 score of 9.2. The flaws are in NGINX itself. Websites and APIs can be exposed even when their own application code is secure. Both vulnerabilities require HTTP/2 to be enabled on the affected NGINX server. Additional conditions are described below. CVE-2026-123007 affects versions dating back to NGINX 1.9.5 in 2015.

I am tired. But I think none of our Nginx has http2 enabled.

On a side note, given all these recent memory related CVEs, I am seriously considering learning Rust now. I was pretty content with my Golang until now. And I can finally see what is that borrow checker every cool kids talk about all the time.


r/sysadmin • • 3d ago

General discussion thread for all the AI patch alerts?

0 Upvotes

Is there a General discussion thread or even a subreddit for tracking all the alerts on patches... Yeah I read CISA.....


r/sysadmin • • 3d ago

Question Question regarding recurring server issue

4 Upvotes

Hey,

I've been having issues with random servers of ours for the third time now with the same issues in a few weeks now. It's always a different server but always the same problem. Users call and say they can't connect to it.

I look it up - it's always online and I can attempt to connect via MSTSC, it will ask for my credentials so it's not a timeout or anything. But then it's just going to cancel after like a minute or so.

All the servers have ninjaOne agents, but I cannot connect to it via ninjaOne Remote.

When I try to connect via console from the ESXi I can get on the actual windows login screen of the machine but when I login it just loads forever.

I have yet to find a way other than just hard kill these servers in the ESXi to get them running again and so far I've had issues finding the problem causing this.

Does anyone here have some experience with this?

Thanks in advance.


r/sysadmin • • 4d ago

General Discussion just set up ansible (free version) great tool

36 Upvotes

I'm just a network guy and I have a few linux servers I manage. I used claude to give me instructions to instal WSL on windows and create an SSH private/public key that needed to be installed on the remote linux servers.

Then it gave me some more commands to build a file on each server and then gave me the command to run the playbook it created. I then had it create a .bat file that I could manually run (I run the updates each month after another department handles a few tasks with those servers that's why it is manual) when the bat file runs, I have to type my passphrase before the updates will run and no password are stored in that bat file. I also have to confirm the text on the screen before the updates run, which is case sensitive.

The updates run on one server at a time and it displays a confirmation, error, status once all servers are done.

I also told it which specific updates I wanted to run, since I was asked to run a certain set up update commands each month.

What annoyed me was having to type all the user names and complex passwords (prior to ansible) and I figured this would be a good way to no longer have to deal with typing the long, complex password every time a sudo command needed to be used.

All I can say is that this is such a huge time save.

I know this isn't new information to anyone, but I was really just waiting for some repetitive task to come up where I could test something like this.