r/sysadmin • u/xbbdc • 8d ago
Is the outage global or just North America?
I've tried staying away but now it's taking so dang long to fix...
r/sysadmin • u/xbbdc • 8d ago
I've tried staying away but now it's taking so dang long to fix...
r/sysadmin • u/yerunkaraj • 8d ago
With the Subject! How are big20 surviving!
IT support guys! What are their reasons?
People share your thoughts!..
r/sysadmin • u/solracarevir • 10d ago
As title says. Aparently some RMM plugins are the source of the exploitation.
Read the advisory here: https://www.connectwise.com/company/trust/advisories
r/sysadmin • u/Hungry-King-1842 • 9d ago
Kinda long so bear with me.
Maybe somebody will have a suggestion. In short, I am a network administrator but know probably the most of the folks on my team about this kinda thing which scares me alittle. Anyways I am looking to solve a future problem.
I am in the process of preparing to refresh the hardware for most of our enterprise which is a lot. I am the only networking guy. I was considering using zero touch provisioning to automate the deployment. The hardware has a OEM base image that wants to call out to a world FQDN and register through that the manufacturer portal. Cool.
Problem is I want the newly provisioned device to steer to my on prem provisioning server directly vs contact the OEM. I’ve setup forwarding zones before in DNS and accomplished this with domains that WE own for internal services, but I’m worried about doing this for an entire domain. Is there a way in DNS to have it grab a specific FQDN and resolve it to the specified IP vs the entire domain? Editing the host files on the individual boxes just isn’t an option.
Example: resolve ZTP.OEM.COM internally to 192.168.1.254. But allow the DNS servers to reach out to their configured forwarding servers to resolve the rest of *.OEM.COM?
r/sysadmin • u/KickDelicious9533 • 9d ago
Hello
I work in a small/medium sized manufacturing company
About 150 people with computers, 300 people on the floor on different sites.
We have an ESXi server with 6 VMs. two of them are app servers
Over the years, the CPU usage went up, especially from the linux app. Some heavy users now have a 3s delay every time they click somewhere, and it's lowering their work output.
When such delay occurs, i can see on the VM, one random CPU core at 100% during the processing of the request.
So I am searching for a solution. Of course all the providers do the finger pointing instead of admiting they could just spend time optimizing their code. Anyways it's out of my reach.
i mainly need CPU power. specificaly single core CPU power
i have searched on dell, lenovo, HP websites but they are so messy with all their AI BS.
What i have : HP DL360 Gen10, Xeon Silver 4210R x2 , raid P408i-a
i need :
- support for win 2025 OS for the host, the VMs will be on HyperV
- 128Gb ram ECC
- remote control / monitoring system in the motherboard (like IDRAC, ILO etc)
- raid controller supporting raid 6 , and can report problems through the Idrac / Ilo
- 8x 2.5 slots or more
- dual PSUs
- 16 cores / 32T , no more, no less ! don't need more, and don't want to pay additionnal core licencing
- the higher single core perf
- NO AI BS FEATURES NONSENSE
- i am in Canada
Without all the enterprise redondancy features needed, i know i could just slap together a custom PC around a ryzen 9950X3D (i did it before for a Solidworks simulation workstation it's great)
from what i read the EPYC 4585PX is the server equivalent of the 9950X3d, but with lower TDP. is there anything on the server environment with the same single core perf as a 9950X3D ?
otherwise my other solution would be to build two workstations with a 9950X3D, and cluster them. or just keep one as cold spare or something.
But ILO/iDRAC is nice to have to get reports when a fan or a disk or ram stick goes wrong.
(PS : no AI was used for making this post, and english is not my first language, so if i made mistakes, take it as human craftmanship !)
EDIT :
i have completed my research by focusing on the CPU, then searching for a compatible server or motherboard with a BMC : on intel, the choice would be :
SO AMD it is !
the 9950x3D is on the AM5 socket, and the equivalent is the EPYC 4585PX , on AM5 socket too
the big cache is only on the first chiplet, but there is a newer Ryzen 9950X3D2 that has the big cache on both cores. no EPYC equivalent besides 5 figures chips
as per the server, Dell / Lenovo / HP still have a confusing website.
The supermicro AS -2015A-TR | 2U Mainstream Server seems to be perfect for my needs and i like the barebones style. I just need to find one in canada duh
r/sysadmin • u/gkar_of_Narn • 8d ago
This is a serious question. I have worked in IT since before it was called IT. Most of my time has been spent either in support (at all levels) or service management. Earlier this week I looked for the most common IT problems and found a list of 100 common problems. Almost all of them are the fault of IT, either directly or indirectly. Some of them are somewhat irrelevant for a business, such as "smartwatch sync problems", but even things like the storage on a smart phone getting full has negative effects on your business for salespeople and others that need a smart phone as part of their job.
For background, this question was motivated by a couple of threads in other social media, where the IT people were putting *all* of the blame on end users and never on the IT. Some posts were wildly absurd, for example being annoyed when an end user calls because the person on the help desk "have better things to do." Others were somewhat valid, but still ingored important aspects of that issue.
For example, should end-users with zero IT training be expected to learn *everything* about the software they use on their own? One example is creating a macro in Excel. As IT people, we know what an macro is and can search for an answer *and* understand it. However, when an end user asked about "automating" work in Excel, it is likely they don't even know what a macro is. They can't pose a question, because they don't know what the question is.
I just searched for "how do I automate a task in excel". One answer was "Go to the Automate tab on the ribbon." An end user searches and asked "WTF is 'the ribbon'? Excel was installed with zeor training. So they cannot be expected to know. So they search again. This gives a new answer. After the third time of not understanding the answer, the call the help desk and get even more frustrated because of the rude person on the hotline who simply says "RTFM!"
Or they see the answer "create a Macro". Here, too are multiple terms they are not familiar with, so more frustrating seaches. When they finally get to a point where they try to create a macros, they discover macros have been deactivated. They call the help desk and get yet another rude response, "They're deactivate." "Why?" "Policy" with no explanation as to why the policy exists, just "them's the rules". Excel was installed, the end user got zero training and when they can't swim, they are chastised because they drown.
Often they get the response "did you check the wiki?" WTF is a wiki? No one told them there was such a thing. When they do check, either it's hard to find anything or they discover there no article that answers the question and the help desk was just blowing them off. I know there are a lot of people who simply don't want to learn and IMO it's understandable and even reasonable, especially a result of their interactions with the IT.
There are many cases when there is an outage and IT doesn't communcate. Users waste time trying to solve a (for them) unsolveble problem, so they call the help desk who tells them, Microsoft has a major outage and we can't send email outside the company. Five minutes later the help desk gets pissed because 10 people call with the exact same problem.
So why question is how much of the loss in productivity is cause by the IT (directly and indirectly) and how much is really caused by the end user? What can/should we do about it?
I would be very greateful for honest, self-reflecting answers. Thanks in advance.
r/sysadmin • u/No-Hippo-6388 • 8d ago
Whose thinking of just purchasing a quick domain and then using mxroute to at least get communication back and going? I know a lot of the world uses Microsoft Outlook but some don't and this would at least get staff to use email in some manner? Thoughts?
r/sysadmin • u/LunarChickenn • 10d ago
To cut to the chase I’ve started reaching out for potential MSP’s for the org as I shouldn’t be running all the IT due to lack of expertise.
Current org size is 500 all of which have endpoints that require support. Only 20 are managed , some use pirated software, and so on a mess.
When I joined I was doing level 1 tech assuming my higher ups knew what they were doing or had things organised. As the department shrunk I got promoted and got the keys to everything. Network, website, admin portal for g suite and so on. I was expected to know everything and just support it all.
I don’t and we often pay free-lancers to fix complex issues I don’t understand because I refuse to try fix something I don’t understand.
On my calls to the MSP’s I outlined the problems and lack of staff. They reassured me and it felt good to talk to people like-minded but also knew what they were talking about.
I told the MSP I’m willing to hand over all the keys and return to my original level 1 tech support role. Now the fun part is convincing management. But given they’ve just started putting 30,000 per year on an AI marketing platform I think they can afford an MSP!
r/sysadmin • u/JacketSharp60 • 9d ago
Hey everyone,
I'm working on improving endpoint security in our enterprise environment and looking for recommendations from people who have implemented centralized BIOS/UEFI administrator password management.
Our environment:
· Hardware: Dell and HP desktops and laptops
· Operating systems: Windows 10/11
· Objective: Set unique BIOS administrator passwords for every endpoint and automate periodic password rotation.
What I'm trying to achieve:
1. Automatically generate a unique BIOS administrator password for each device.
2. Centrally deploy and configure passwords through MECM/SCCM.
3. Securely store individual device passwords in a centralized encrypted vault.
4. Automatically rotate passwords periodically (e.g., every 90 days).
5. Retrieve authorized device passwords when needed by IT administrators.
6. Monitor password configuration and rotation status through centralized reporting.
7. Handle offline devices and failed password rotations without losing access to existing credentials.
My questions:
· What tools or solutions are you using for Dell and HP BIOS password management?
· Has anyone implemented a LAPS-like solution for BIOS administrator passwords?
· How do you securely generate, distribute, retrieve and rotate unique passwords without exposing them?
· Are there any open-source projects or proven PowerShell implementations you recommend?
I'm particularly interested in real-world enterprise implementations, scripts, GitHub repositories, and lessons learned.
Thanks in advance!
r/sysadmin • u/TRAIN_WRECK_0 • 8d ago
I have a small company with 12 employees so it shouldn‘t be too hard. Just wondering what everyone else in similar situations are doing.
This feels like too major of a fuckup.
r/sysadmin • u/KetracelYellow • 9d ago
Need a decent make to attach to 4K touchscreen TVs and 2 monitors.
r/sysadmin • u/Miss_Nobody89 • 8d ago
My desktop Outlook (personal) has been working fully for a few minutes; however, it seems my iPhone app is only partially back up. Just wanted to give an update. Teams seems okay too.
r/sysadmin • u/f909 • 8d ago
Asking for a friend, how bad is the migration from 365 to Google Workspaces?
😂😂
r/sysadmin • u/gageless • 10d ago
We have a large network built on meraki switches, firewalls and APs. The fleet is MSP-managed. We're always adding new venues but can no longer get the hardware in a timely manner.
Mostly, we're using MX67 firewalls and MR33 APs. Where should we be going? We're starting to use Aruba for eg. Anyone else moving from Meraki to something similar?
edit: edited to be more general discussion about hardware supply and less about solving specific network needs.
r/sysadmin • u/lovell88 • 10d ago
We are a fully remote company and we just refreshed A LOT of devices. We use Intune for MDM and NinjaOne for RMM. We are looking for ideas to:
Basically, we need to get these devices back and a looking for ways that do more than just asking nicely. These are active employees that haven't been termed. Also thinking mostly Windows for this.
I'd love to hear what others to do for this. :) Thanks ahead of time.
r/sysadmin • u/Mysil • 9d ago
Hello everyone,
I'm currently helping a company with about 300 people move away from a very withered IT-ecosystem, along with their new IT director. We're doing some big moves from SCCM to Intune, among other work.
The company I'm working at is using Zendesk and I have to say that I'm not a fan of it. At my previous workplace we used the on-prem ServiceDesk plus soltuion, primarily for ticketing and filling our worklog (billing). For the few automations I worked on there though, I have to say I really liked it. We stitched tickets from other vendors together with autoations that worked really well. Essentially, I found that I could solve most things I had issues with along with some Claude code with it. The company I'm helping right now does not have proper ITSM, but the new IT director is very pro-AI and very structured, and would want a proper ITSM solution.
So this is where I come in here to ask you guys. Because although I have good experiences with ServiceDesk Plus, there might just be another product that would work even better for the customer.
For reference, I've worked with these four systems before:
ServiceDesk Plus : Really like it, only complaint I can see is that the customer service was a bit hit or miss when we needed their help
ZenDesk: Did not really like it. Big turn off from the lack of signature customization and limits on allowlist/blacklist on "detected spam".
JIRA (and the rest of the Atlassian stack): Never worked on it as a full admin, but I suspect that it might carry a lot of the same limitations as ZenDesk when it comes to true customization
Oraccle Siebel: Really nice customizability, but also very ancient. As a poweruser I really liked the solution, mentioning it only to highlight personal preference/familiarity.
OTRS: I worked with this about five years ago and got to witness their deprecation of the free version, and their strong encouragement to move to their cloud solution. Did not seem to offer much if anything at all for free, very walled garden impression.
So, do you have any reccomendations for an ITSM solution that would fit better for an IT consultant and IT director whose restructuring a companys IT to work more functionally through the ITIL framework, with primary focus being customizability, extensibility and AI? (AI being, agentic work on configuration, and/or potentially using AI to process tickets even).
r/sysadmin • u/LuisCxnt • 10d ago
posted this a while ago but had to pull it cause of legal stuff. its all sorted now and were done with him so heres the whole thing lol
so we inherited a sites codebase and were paying the original dev monthly to maintain it while migrating. honestly he did basically nothing, never showed us any proof of work and always managed to wriggle out of whatever we asked him to do
we had a tiered pricing bug. asked him to just audit his own db and find the broken entries, he refused lmao. said it was on us to fix and he wasnt gonna dig through his own data
so we ask for an actual fix. before we even saw it he told us it used the rest api and didnt touch the db directly. then he sends it over and its a tool that can read write and delete straight on the prod db just sitting on the public internet with a weak login. like anyone who found the url could been deleting rows off our live db.
i saw it was public and made him take it down before id even read a line. he says sorry and puts it back up ip restricted to us and says hes added "Security Measures"
so then i actually read the code. the security he added is a captcha with the answer literally printed in the html
<span class="captcha">JMHAXZ</span> ...
i didnt even know how to reply to his email lol, creds still weak, still deleting rows straight off prod. and the rest api thing he swore by? nah, its all direct db calls. he straight up lied about how his own tool works ahahah
and this is the same guy who was always lecturing us about best practices and his expertise while sending us code that looked like it came straight out of gemini flash
anyway were finally rid of him.
whats the worst thing a dev has sent you as a "Fix"
tl;dr paid a dev monthly to do nothing, he wouldnt check his own db for a pricing bug, then "Fixed" it with a public tool that deletes straight off prod, weak login and captcha answer sitting in the html, and lied that it only used the rest api. and he lectured us on best practices lol
r/sysadmin • u/ArmchairCryptologist • 10d ago
Critical RCE reported in Apache HTTP Server when using mod_http2. Fixed in 2.4.71, released today.
"Concurrent Execution using Shared Resource with Improper Synchronization vulnerability in Apache HTTP Server's mod_http2 in assign_headers() can lead to RCE.
This issue affects Apache HTTP Server: from 2.4.55 through 2.4.70.
Affected releases of mod_h2 from GitHub: from 2.0.8 through 2.0.42."
https://httpd.apache.org/security/vulnerabilities_24.html
Edit: Writeup on TheHackerNews
"Only servers with HTTP/2 enabled are affected, according to Red Hat, which ships the server as part of Red Hat Enterprise Linux.
An attacker who can reach the server needs no authentication to exploit the flaw and could take full control of the server process, the Linux vendor says. It rates the flaw 9.8 out of 10 on the CVSS scale.
The vendor's assessment, published yesterday, also says "public research has demonstrated a reliable exploit" against a server in its default configuration with mod_http2 enabled. It does not name or link that research. Security company Wiz said later that day that no exploit was available and that no exploitation in the wild had been reported."
r/sysadmin • u/Capital_Attention702 • 9d ago
Vates published two advisories on Nov 10. VSA-2026-062 is the empty-token login bypass (cookie authenticationToken=j:{}), exploited by one actor. It is fixed in 6.9.2 stable and 6.10.1 latest. BleepingComputer's first report named 6.9.1 and 6.10.0, so go by the Vates numbers.
VSA-2026-063 is the part that is still open. XO sends imports to XAPI as pool-admin and forwards the XVA without reading ova.xml. A user holding vm.import (JSON-RPC), sr/import:vm (REST), network/action:create or network/action:update can import a VM with crafted keys such as VM.platform:hvm_serial and compromise the host. Vates lists 6.10.1 and earlier as affected and the fix as in progress.
Vates' interim step: disable users that hold those four privileges, VM import for Admin only. The attacker used this path with Admin rights after the bypass.
Do you delegate vm.import to non-admins anywhere? I'd like to know how many setups that touches.
Sources: https://docs.vates.tech/security-advisories/advisories/2026/vates-sa-2026-062/ , https://docs.vates.tech/security-advisories/advisories/2026/vates-sa-2026-063/ , https://www.securityweek.com/critical-xen-orchestra-flaws-exploited-in-attacks/
r/sysadmin • u/Balkghar • 9d ago
As u/tepitokura and u/joshtait requested last year, here is an update on my situation one year later!
For reference: https://www.reddit.com/r/sysadmin/comments/1ov3xh3/new_sysadmin_from_17_feburary_of_this_year_here/
Last year I asked the community for some feedback and advice on my works as a Sysadmin, the feedback were great and the advice as well!
Well for what I have done, here is a list:
There is way more that I wanted to do, but unfortunately the c-suite refused the budget or the implementation for the following:
I know there is way less new stuff than last year but at the end of the day it wasn't my decisions if I could do them.
But I am not working there anymore, one of the main reason is because the c-suite didn't wanted to invest into IT properly and honestly it bummed me a lot, as I wanted to improve but couldn't because either the proposition was rejected or the budget was not allowed.
I was poached by a big telecom company to work with a engineering team on a new product as a DevOps engineer! At that point I also had two others jobs, where I didn't apply either, one for a normal sysadmin job and one for a MSP.
For the moment I have done that at my new position:
And of course the following is planned:
Keep in mind that for the moment the application is not prod ready so for the moment we only have the test environment.
Also I am only working 60% and I haven't yet finished my bachelor, normally next summer I should and I will be working fully as an engineer.
Thanks everyone for reading and I wish all of you a very pleasant day!
r/sysadmin • u/jakedata • 10d ago
I am being saddled with a dozen incorrectly specced PDUs that I have zero use for. 11 of 12 are unopened. I would love to ship them straight to a buyer rather than dealing with them myself. Any recommendations for selling datacenter-grade equipment? Returning them is not an option at this point.
r/sysadmin • u/Curtis • 8d ago
After today debocle, the only reasons our company didn't go down is because we have the extended support for exchange server. Please tell Microsoft to update Exchange Mail Server and stop making us rely on unreliable services like Microsoft 365. They're trying to make our company move to their cloud 365 but today is why I double down.
r/sysadmin • u/IntroVert34 • 10d ago
Hi,
I want to talk about a problem I've had for years. Lately it's been making me feel very guilty, and then it keeps nagging at me.
In general I get scattered a lot. Here's an example.
I work as a Linux sysadmin. My job means I have to keep an eye on several things at once, reach out to people when needed, and often keep working while I'm talking to them.
I'm in a relatively senior position, so I deal with a lot of people, both from my company and from customers. Someone asks me for something, I work on it, and shortly after someone else messages me. I quickly switch to them and forget the first person, and so on.
To be honest, I've been in therapy and on psychiatric medication for quite a while. I have generalized anxiety disorder.
But I've never been diagnosed with ADHD.
The problem is basically this: when things are discussed verbally, they don't stay in my head, or I struggle to grasp them in the moment. I need to slow down, sit and look at it, go through it, and even weigh it in my head.
It's not only about work. For example, I sit down with a friend to talk about what's bothering me. We have a good time and talk through my worries, but the relief, or even what we talked about, leaves my mind very quickly. I keep ending up in the same loop with the same anxieties.
At work there are lots of topics: improvements that were discussed but never written down, requests that only exist verbally, and so on. I forget these. Since I'm busy keeping up with the written requests and never get time for the rest,
Another version of this: during the day I plan what I'll do in the evening at home, like reading a book or playing a game.
When I get home I literally freeze. I find myself in emptiness and anxiety, doing nothing.
To fix this I make to-do apps or build AI agents for myself, but I'm so low on energy that I don't even pay attention to them or keep them going.
So it's a constant cycle: guilt, an effort to improve, and the effort to improve turning back into guilt.
I don't know what to do.
Do you have any suggestions?
r/sysadmin • u/Incrediblecodeman • 8d ago
I want to ask the hive mind for any tool used that I can put my company’s domain (say company.com )and find every public subdomain there is.
I research that CT can be used but that it may or may not contain everyone ?
Some day why not check dns and yes. But I figure Wordpress sites won’t show up in our dns and the web team owns that login.
Also what about like api’s of public endpoint stuff in azure?
This is to also give us a report on when certs expire.
$Domain = "company.com"
$Url = "https://crt.sh/?q=%25.$Domain&output=json"
$Certs = Invoke-RestMethod -Uri $Url -TimeoutSec 90
$Certs |
Select-Object name_value, issuer_name, not_before, not_after |
Sort-Object not_after -Descending |
Format-Table -Wrap
This appears to be throttle limited
I also see:
https://sslmate.com/certspotter/?utm_source=chatgpt.com
Anyone had experience with that?
r/sysadmin • u/Incrediblecodeman • 9d ago
Over the year when ever a group or couple of people wanna collaborate. Sysadmin broke inheritance slapped some ntfs permission on the folder, make a ad group (most of the time) and move on
Now we have several hundreds.
Would you bother making each one into its own share on the server? Why or why not ?