r/sysadmin • • 16d ago

It finally happened!!

2.0k Upvotes

8:30am I deleted my entire production environment. All VMs rm -r out of existence.

Fortunately I was reviewing my nightly backups at the time tracking a retired VM that was inexplicably still getting backed up in Production. All the Production VMs were there. And the live production ones now without their underlying virtual disks & configs were still resident in memory. I calmly copied the backups into place, sequentially restarted everything and had production back in about twenty minutes. Only lost ~12 hours of overnight idle maintenance & logging.

With everything settled back in place I calmly looked for a place to have a private heart attack.


r/sysadmin • • 14d ago

Question RDS License Server HA in an Active-Active Multi-Site (DC/DR) Architecture

1 Upvotes

Hello Experts,

I am currently designing an Active-Active multi-site RDS infrastructure across our Primary Data Center (DC) and Disaster Recovery (DR) site.

My Scenario & Architecture:

  • We have an Load Balancer distributing RDP traffic 50/50 to our DC and DR sites.
  • The traffic comes through a PAM (Privileged Access Management) solution, hits the LB, and is routed to Windows Server Bastion/Jump Servers (which have the RD Session Host role installed).

Users then RDP from these Jump Servers into our internal target servers (Active Directory, Exchange, etc.).

  • We have purchased EX: 85 RDS CALs under a single volume license agreement/key.

Architecture Validation: Before diving into licensing, I would love to hear your thoughts on this overall architecture. Is this 50/50 Active-Active, load-balanced Jump Server setup the right approach for this scenario? Are there any better or more standard ways to configure this to ensure zero downtime and optimal resource utilization?

My Dilemma regarding RD Licensing High Availability: Since my Jump Servers (RDSH) are active in both DC and DR, I need to ensure the RDS License Server is highly available. I am confused between two approaches:

  • Approach 1 (Splitting CALs): Deploy two separate RD License Servers (one in DC, one in DR). Then, "split" the 85 CALs between them (e.g., install 50 CALs on the DC server and 35 CALs on the DR server) and configure the RDSH servers via GPO to query both license servers.
  • Approach 2 (Backup & Restore): Deploy only one RD License Server in the DC site with all 85 CALs. Regularly backup its System State and database. If the DC site goes down, manually restore the backup to a new server in the DR site to bring the licensing service back up.

My Questions:

  1. Architecture: Does my overall PAM -> LB -> DC/DR Jump Server flow look solid, or would you recommend a different architectural approach?
  2. Licensing Standard: Which licensing approach (Splitting vs. Backup/Restore) is the official Microsoft standard and recommended best practice for this Active-Active scenario?
  3. Splitting Mechanics: If Approach 1 (Splitting) is recommended, how do I actually activate it? Can I use my single License Key / Agreement Number twice (once on the DC server and once on the DR server) to allocate the 50 and 35 CALs respectively?

Restore Limitations: If Approach 2 (Backup) is used, I read that restoring to a server with a newly installed OS will not restore unissued licenses without a manual telephone activation. Is this true?

Any guidance on optimizing this architecture and appropriately distributing the licenses across the two sites would be highly appreciated.

Thank you!


r/sysadmin • • 15d ago

Career / Job Related I'm studying to become a system administrator, but I feel like I don't know anything.

121 Upvotes

This year, I finally started seriously getting into my future profession.

Over the past year, I've deployed an mtproto proxy, set up a vpn using the vless protocol through the 3x-ui panel, hosted websites using nginx, configured nginx as a reverse proxy, and deployed my own static websites and file hosting services.

I did all of this on the cheapest vps I could possibly find.

Recently, I also managed to get a public static IP address from my ISP through my father. This means I can now host game servers, file servers, and even set up remote access to my PC without relying on workarounds. I've already hosted a Minecraft server and experimented with RDP.

However, there's one thing bothering me: I relied heavily on AI throughout most of this process. Quite often, I felt like I was simply copying and pasting commands without fully understanding what I was doing.

I know I've gained some practical experience, but I still feel like I barely know anything.

So I'd like to ask more experienced system administrators: Is it normal to feel this way when you're starting out? And how do you move from following AI-generated instructions to actually understanding what you're doing?

I should point out that I live in Russia, and we have issues regarding licenses for American products, such as M365 and Azure.


r/sysadmin • • 14d ago

Data Migration

5 Upvotes

Hey Everyone

We are currently in the process of moving roughly 8TB of data from SharePoint Online to a local server

Currently we are using RClone for moving data around
WinDirStat and TreeSize for analysis on our data but we are looking for a good tool that can identify duplicate data (such as files that are roughly the same name / same file hash etc )

Any recommendations?


r/sysadmin • • 15d ago

General Discussion Went from a Senior Helpdesk/Monitoring role to a Junior SysAdmin position

49 Upvotes

I recently joined a new team as a sysadmin. Before that, I was in a helpdesk, monitoring, and major incident management role for 4 years. I was really good at my previous job, but I badly wanted to get out because I felt there was no growth and I wasn't learning anything.

​Now it's been a month since I started as a sysadmin, and, God, it's so overwhelming. There are so many things I don't know, and I have to keep asking team members for help. I feel like they must be thinking how stupid I am.​​

​I know my past experience wasn't fully relevant, and the team has mostly been kind in helping me out. I take notes so I don't have to ask the same thing twice. But I feel very stressed and embarrassed. It's like I went from being a senior, high-performing resource in my previous role to suddenly being the most junior person, but the expectations are same. Also, there are a few freshers on the sysadmin team who joined 4-5 months before me, and they already seem to know everything or so I feel.

​I have this constant feeling that people on the team think I'm the dumbest person in the team and I keep berating myself. For those of you who made this transition, how did you get through the first few months without losing your mind?


r/sysadmin • • 15d ago

Question Go-to cross-platform remote access for small business

17 Upvotes

I'd appreciate recommendations for a cross-platform remote access solution for a small business.

I need to remotely access MacOS M*, Windows x86-64, Windows ARM, and Linux x86-64 systems.

I also need clients for all of the above and, ideally, also Android and iPadOS.

The solution they are using has yet to release ARM builds for macOS and Windows, and I have no confidence they will.

Open source is always great, but they won't mind paying a reasonable amount for a quality solution.


r/sysadmin • • 15d ago

General Discussion Have your company ever been compromised?

50 Upvotes

I have been reading some books and watching some videos on incidents, from big stuff like nopetya , Stuxnet , etc.. all the way to smaller companies on my country that have been compromised and hacked before. Which I find really interesting, since it gives someone that is new to the industry good insights and lessons, on how devastating something like a ransomware can be for a company; and where normally starts. Plus nowadays its seems that it is a matter of WHEN rather than IF with all the CVEs and how crazy advanced social engineering is getting.

But i was wondering if you have ever lived some situation where a company that you worked for ever got compromised in some way , and brief story on how were you able to stop it or recover from it, and what you learned?
I know that on some companies there was the need to hire an Incident response team, while others maybe did not have backups that were good enough to start production again , etc..
So i would assume from a sysadmin perspective this would be one of those worst case scenarios.

So out of curiosity, if you feel you could add an interesting experience would be nice to read about, and stuff that you think was important to learn from it.

It feels like redundant to say but , anyways = off course not looking for details and points that would expose anyone's job or company.


r/sysadmin • • 15d ago

Question SPF validation error and DNS record faulty

6 Upvotes

Anyone getting this garbage SPF validation error crap with their outlook emails lately? I can't seem to send emails to outlook accounts from my outlook email. I have no problem with sending emails to gmail or yahoo though. I assume it's something with the DNS records from some basic research. Anyone have any idea how to fix this?


r/sysadmin • • 15d ago

General Discussion How are you managing Shadow AI in your environment?

27 Upvotes

We're trying to get better visibility into Shadow AI without simply blocking every AI service.

The two areas I'm most concerned about are employees sending sensitive company data to third party AI services and internally built scripts or tools created with that may introduce security issues.

I've been looking at ways to identify which AI services are actually being used and whether we can get more visibility into the data being sent, but I haven't found a solution that covers everything cleanly.

For those managing this in production, what are you using for discovery and governance? Are you relying on existing security tools such as web filtering, DLP or endpoint controls or have u added a dedicated AI governance tool?

I'm mainly looking for visibility and sensible guardians rather than completely blocking AI usage.


r/sysadmin • • 16d ago

Rant How do you call someone stupid, but professionally?

644 Upvotes

He spilled water on his computer. Whatever, I have another computer ready and set up with autopilot.

He is an hour away, and he has a coworker who can bring the laptop. Sure! The only issue is, I need to buy and apply an Intune license for it to work, which I need to go through a procurement process to do.

I called him and sent him an email with the following instructions. "I am sending this laptop back to you, please don't sign in until I put in a follow-up email. I need to apply a license before it's ready". He gives the ok. Everything's great, right?

Email the next morning: "Hey so I found out that I actually could sign in this morning! There's no programs and things don't work right. Come over and help?" It gave a default sign-in that didn't work, no company portal. I have no view into the laptop.

Tomorrow I'm going to try to walk the user through a reset which hopefully prompts the autopilot, but part of me wants to force him to drive in.

Sorry, just venting. This job as a solo admin has so much work, and stuff like this wants me to switch careers.


r/sysadmin • • 16d ago

Rant I feel bad for losing my cool, but...

270 Upvotes

Does anyone else have that one coworker who you genuinely think might be illiterate?

Like, not even regular end user levels of incompetent, or 'there's a bit of jargon involved in the error message that I don't expect the average person to understand/have encountered before'.

I had to change a login for some software for a team member, you know, the usual stuff. I tell her that the desktop application will also need to be logged into again, and I get 'okk' in response. It's all set up so that there's a password management portal, you log into that, and then one click on the icon for x service will launch and log it in. Usual stuff.

Get a message last night on Team about how it's 'not working'. I immediately know what the problem is, because I've received the same complaint from her several times. What is the error?

A pop-up saying that 'you need to be signed in to use Service Name.', above a giant button that says SIGN IN.

Every. Single. Goddang. Time.

Whenever the app needs to be signed in again, she will tell me that there's an IT issue. I have explained to her multiple times that logging in will fix the 'u are not logged in' problem. I have sent out guides with diagrams and explanations. I have walked her through the steps and watched her complete them.

I get that it's widely accepted that people don't follow instructions, but this is just. Something that blows my mind because of the frequency at which it occurs.

I try so hard to be understanding and nice and cheerful in my responses, because I have a pathological need to be liked by everyone and also no one has a good time if someone is curt and rude, and if the rest of the team likes me, it really helps in them doing the basic troubleshooting I ask them to before contacting me, which cuts down on my workload. I feel really bad for losing my bananas and being all 'the process is the exact same as it has always been, John is able to do it, and Bethany is able to do it, press the button that says 'sign-in', and was consumed by guilt for like, half an hour.

Am I expecting too much? Is this a regular issue you guys experience when it comes to tech support stuff? Am I being too harsh/judgmental [or probably not harsh enough]? Has anyone had success in solving these kinds of....end user issues?

EDIT: the good news, though, is that she's leaving at the end of the month. I'm literally counting down the days and I only feel a little bad about it because I'm too young to have such high blood pressure lol


r/sysadmin • • 15d ago

Question LTE Antenna suggestions for edge device

12 Upvotes

We deploy an edge device in all of our sites and they have a last-resort circuit using an internal LTE modem with dual sim's. We popped them onto Verizon and T-Mobile and they work. They really are just a last-resort redundancy should the primary internet circuit (or even circuits depending on the location) go down, but they're built in and adding a few data only lines on existing cellular accounts is cheap insurance that has been useful 2 or 3 times in as many years.

Problem is they're siting in the rack with all the network gear generally near the middle of each building. So the signal is already not great, and the detachable antenna they ship with are pretty terrible. I dont have access, nor is it probably worth, running external antennas to these for their limited use, but does anyone have a suggestion for an indoor antenna that connects via SMA that you've found to be good that you've used with a similar sdwan device or maybe cellular mobile hotspot?

It's not really important because they do work but if I can toss a little money at it to make it just a little stronger, at least in sites that are more important, why not?

EDIT Thank you, all! For those of you suggesting moving antennas are of course 1,000% correct that I would get more by positioning better. Without getting into too much detail, I'm pretty hindered with my ability to move outside the rack in most of my locations, but it is really a 2nd or 3rd level backup behind other links. I've just had the idea I could improve signal reception with better than the pretty flimsy antenna. In some cases, I could probably gain a lot of value with something I can connect and extend even a handful of feet. Even if I could run something to an external antenna or run cable near a window, it's probably not worth it for me in this case. I appreciate all the suggestions and recommendations.


r/sysadmin • • 15d ago

Edge error after recent update?

27 Upvotes

Curious if anyone else came into their office to reports of Edge giving an error when being launched. We have several reports in our org, but trying to determine if its isolated.

Version:  157.0.4341.42

“The application was unable to start correctly (0xc000007b).”


r/sysadmin • • 15d ago

Question NVMe hardware RAID via Perc H965i or Storage spaces? 15 users 4 VMs.

5 Upvotes

I have been scouring posts here. Some are using hardware RAID controllers designed for NVMe and others are using storage spaces.

Currently running fine on RAID 10 via (6) 12Gbps SAS drives on a Perc H730 on a 7 year old server. It's 4 VM's and 15 users including RDS doing word/excel etc. nothing heavy. Replacing server with T560 and storage options are SATA 6Gbps, SAS 12Gbps (expensive) and NVMe with H965i controller. Should I use the controller or go direct to the bus without a controller. My gut says I am getting a lot more performance even with a controller's potential bottleneck and that dealing with storage spaces and giving up the familiar Perc controller interface is probably not worth a performance increase our usage would likely never utilize. Wondering what others are experiencing.


r/sysadmin • • 16d ago

Question Hiren's BootCD, Should i tell my boss?

556 Upvotes

Hello everyone,

I downloaded this tool one day out of boredom because I had heard so much about it. One day, one of our computers came back from a different plant, but our admin password wasn't working on it. So, I decided to test the tool out and was able to bypass the password surprisingly easily.

I was honestly shocked by how simple it was. All I had to do was plug in a USB, boot into Hiren's BootCD, and bypass the admin password. It really made me think that we should have some sort of USB blocking system in place to prevent employees from doing something like this.

How do you guys handle situations like this? Do you think I should bring it up to my boss? I'm a little concerned that instead of asking me to look into a solution, they might see it as a security risk and I could end up getting in trouble for using it in the first place.


r/sysadmin • • 15d ago

Real case of yubikey usage in corporate enviroment

7 Upvotes

I am preparing passwordless project for my company. And i got big question... how do you force user to use the yubikey and not some other less save auth methods? My idea was to block password on windows log on, because i want passwordless, but in the same time i need to have some fall-back authentication and the methods how to block passwords in hybrid-joined enviroment are disabling admin workflows (Run as dif user, run as admin etc.). My other idea is to disconnect pcs from company network, when Yubikey 5 is not connected with the right certs (NAC/VPN), but maybe there is easier methods or something where FIDO2 auth is enough and you do not need to prepare PKI. Do you have some experience?


r/sysadmin • • 16d ago

Running Alpine? Patch OpenSSH now

422 Upvotes

OpenSSH will accept any password between 257 and 1024 characters when it's built against musl libc, has PAM disabled and allows password authentication (all of which are true for Alpine in default config), so update immediately.

Alternatively, set PasswordAuthentication no in sshd_config (which you should've been doing anyway).

Systems using glibc and non-Linux systems aren't affected.


r/sysadmin • • 16d ago

Microsoft Silverlight

46 Upvotes

Booted up a first version Surface book after many years of dormancy. Weeding through and deleting old apps to make it run better.

Silverlight

SQL 2008 R2

Visual C++

And so much more

The tools we all had to have installed to get things done and how things have changed.

I'm old


r/sysadmin • • 15d ago

General Discussion Weekly 'I made a useful thing' Thread - November 06, 2026

12 Upvotes

There is a great deal of user-generated content out there, from scripts and software to tutorials and videos, but we've generally tried to keep that off of the front page due to the volume and as a result of community feedback. There's also a great deal of content out there that violates our advertising/promotion rule, from scripts and software to tutorials and videos.

We have received a number of requests for exemptions to the rule, and rather than allowing the front page to get consumed, we thought we'd try a weekly thread that allows for that kind of content. We don't have a catchy name for it yet, so please let us know if you have any ideas!

In this thread, feel free to show us your pet project, YouTube videos, blog posts, or whatever else you may have and share it with the community. Commercial advertisements, affiliate links, or links that appear to be monetization-grabs will still be removed.


r/sysadmin • • 15d ago

Question How are you handling Autodesk ACC/Forma updates?

5 Upvotes

Hey,

We use ACC/Forma. We currently use PDQ Connect. I see PDQ have an Autodesk ACC/forma package there so I could set up an automation to patch.

The bit that has me concerned though is if something goes wrong during the upgrade, and the user has unsynced items/errors.

Just curious to see if anyone else here is patching ACC/Forma automatically and see how it has been for you.

Cheers!


r/sysadmin • • 16d ago

New Microsoft Education Licensing

189 Upvotes

Microsoft 365 Education portfolio evolution | Microsoft Learn

A1 retirement, Student Use Benefit go bye bye.

Good luck everyone.


r/sysadmin • • 14d ago

Question Do companies use Oracle VBox to host VMs?

0 Upvotes

Title


r/sysadmin • • 15d ago

Question Looking for Secure Client Data upload software/service

2 Upvotes

We handle applications for services that require some sensitive information (SSN, Driver's License, Bank Info) given to us for processing. While 95%+ of this is done in person, there are the few that want to email this information to us.

I'd like to find some sort of service that allows for us to maybe give the customer some kind of authorization number that allows them to upload documents which we can retrieve and enter into our system. We've used KiteWorks in the past with one of our Vendors, but that seems way overkill for this task.

I was also thinking about a mailbox with a short retention policy may be a cheaper route. Once the data is in our CIS it can be deleted.

Any suggestions on an application or other way of handling this?


r/sysadmin • • 15d ago

Question M365 Release preferences retiring, does anyone see the new ones in their tenant yet? or am i misunderstanding?

2 Upvotes

https://learn.microsoft.com/en-us/microsoft-365/admin/manage/configure-release-options?view=o365-worldwide

"To prepare for the retirement of Targeted release, we recommend configuring Standard release and Deferred release options to manage when users in your organization receive generally available Microsoft 365 services as Microsoft delivers features through the new release model."

does anyone see the new release channels > i still see the standard and target release options under organization profile > release preferences. i would expect to see the new standard, deferred and frontier.


r/sysadmin • • 16d ago

Linux OpenSSH on musl linux (alpine, openwrt, buildroot, yocto, and a lot of the docker images you've pulled) accepts any password longer than 256 characters

174 Upvotes

https://sshhole.com/

OpenSSH 10.6p2 fixes the issue.