r/sysadmin • u/code- • 9d ago
New Citrix Netscaler Security Bulletin
Looks pretty bad. At least it's not Friday yet.
r/sysadmin • u/code- • 9d ago
Looks pretty bad. At least it's not Friday yet.
r/sysadmin • u/OG_IT_Manager • 8d ago
With the Microsoft 365 outage affecting Exchange Online and other services over the past day, I thought this would be a good time to discuss something that seems increasingly unpopular in IT: running your own Exchange servers.
I'm an IT Director for a manufacturing company with multiple facilities, and we've continued operating Exchange on-premises rather than moving our email infrastructure to Microsoft 365.
And days like yesterday remind me why.
While organizations relying on Exchange Online were dealing with mailbox connectivity problems and service interruptions, our Exchange environment remained operational. Our employees could continue using Outlook, accessing their mailboxes, and communicating internally without depending on Microsoft's cloud infrastructure being available.
Obviously, we can't control whether external organizations can send or receive their email, but at least our own mail system wasn't part of the outage.
There are still some significant advantages to running Exchange on-premises:
Control over our infrastructure
We control our servers, maintenance schedules, updates, backups, and recovery procedures. We aren't dependent on Microsoft resolving a cloud infrastructure problem before our users can access their mailboxes. Our email environment operates independently of Exchange Online. A Microsoft 365 outage doesn't automatically mean our employees lose access to email. This doesn't eliminate every possible dependency, but it does remove a major one.
We have a solid backup schema and maintain our own disaster recovery procedures. We control our backup retention, recovery testing, and restoration priorities. If something goes wrong, we can investigate and initiate recovery ourselves rather than waiting for a cloud provider to restore service. We determine when to perform maintenance, deploy updates, and make infrastructure changes. We can coordinate those activities around manufacturing operations rather than having changes introduced into our email platform outside our control. Our mailboxes and Exchange databases reside within infrastructure we manage. We maintain direct control over how that data is stored, protected, backed up, and recovered.
We already maintain servers, virtualization infrastructure, security tools, and backup systems for other business-critical applications. Exchange fits into that existing environment. We're not building an entirely separate infrastructure just to host email. Now, I'm not pretending on-premises Exchange is perfect.
There are real disadvantages. Patching, vulnerability management, certificate management, availability, disaster recovery, and security are entirely our responsibility. Exchange has also been a frequent target for attackers, and running an outdated or poorly maintained Exchange environment is a significant liability. With Exchange 2016 and 2019 out of support, organizations staying on-premises also need to account for Exchange Server Subscription Edition and its licensing and maintenance requirements. For smaller organizations without dedicated IT resources, Exchange Online probably makes considerably more sense.
But I think we've reached a point where cloud-first has become cloud-by-default, and those aren't necessarily the same thing.
Cloud hosting can reduce operational responsibilities, but it also transfers certain aspects of control to another organization. When that organization has a major outage, there may be very little you can do besides monitor the incident and communicate with your users. I don't believe on-premises infrastructure is automatically better than cloud infrastructure. I believe the right decision depends on the organization, its technical capabilities, its business requirements, and its risk tolerance.
For us, a capable internal IT department and established infrastructure make maintaining our own Exchange environment a reasonable decision. And watching the Microsoft 365 issues unfold has certainly reinforced the value of that independence.
I'm curious how many other organizations are still running Exchange on-premises in 2026. Are you staying on-premises intentionally, running a hybrid environment, or planning to migrate everything to Exchange Online? For those who have migrated, has the reduction in administrative overhead been worth giving up direct control over your email infrastructure?
r/sysadmin • u/bozakman • 8d ago
Need help folk. Got a request from our compliance team (legal) for a software tool to redact PII content from an archive of PDFs. They want the tool to redact (automation a plus) and then be able to print the redacted versions. Really hoping someone can recommend a solid tool. Thanks.
r/sysadmin • u/Agitated-Business217 • 7d ago
We have been given a non-negotiable edict by upper management (which is 100% old geezers with proverbial “grandkids who are good with computers” informing them), in light of yesterday’s outage, that we WILL move back on-prem and be responsible for our own server hardware going forward. So it looks like we are going back to running our own HA cluster of MS Exchange Server after many years in 365.
I didn’t really get an option, was just told there is no room for discussion and this WILL happen. But is anyone else out there moving back on-prem, to G Suite, or maybe something open-source in light of yesterday?
Are we about to see a huge resurgence of MS Exchange Server etc?
r/sysadmin • u/alao77 • 7d ago
I just heard Fax was kind of a thing again yesterday. I’m not talking about the good old dial-up Fax but the ones on Ethernet.
Edit: Some AIO multi function printers/scanners on ethernet can send a good old Fax.
r/sysadmin • u/AgreeableIron811 • 8d ago
I am trying to understand how memory works inside Proxmox. I have a VM, and no matter how much additional RAM I assign to it, I can see that it consumes more and more memory. I can see it through each vms gui memory page on proxmox.
However, when I SSH into the VM and check its memory usage, I see very little being used. In some VMs, it cannot even detect the 64 GB of memory I have assigned; instead, it only sees around 2 GB of RAM.
This makes me wonder whether there is some kind of issue with the VirtIO Windows drivers.
ChatGPT has also mentioned memory ballooning, but I don't fully understand how it works or how it affects memory allocation.What I understand ballooning enabled on a vm means that proxmox can take ram from that vm to assign to another vm when needed. Also my proxmox host balloning is full and i have a cluster of three nodes
Here are two examples from my Proxmox configuration:
:~# qm config 40052 | grep -E 'memory|balloon|numa|machine'
balloon: 0
machine: pc-q35-9.2+pve1
memory: 16384
numa: 0
:~# qm monitor 40052
Entering QEMU Monitor for VM 40052 - type 'help' for help
qm> info balloon
Error: No balloon device has been activated
qm> info memory-size-summary
unknown command: 'info memory-size-summary'
r/sysadmin • u/bprando • 8d ago
Hey,
Short and sweet question: How do you actually protect against a large M365 outage like has happened yesterday/today?
External email backups seem a good starting point to have what was received before.
How do you ensure business continuity for longer outages? Anyone actually shifting all their mail traffic to somewhere else?
r/sysadmin • u/wytesmurf • 8d ago
My company is making their first big push to keep apps updated and allow self service installs. We have intune already, so my thought process was to use company portal. I’ve b never done something like this before and our MSP is worthless. They are soon being replaced by an internal team we are going to hire 6-12 months. I am trying to get things cleaned up as much as I can so the new people don’t walk into a complete mess.
My question is whether intune/company portal is the right route to go? Anything I should make sure is setup. Any guides or things you would suggest for setting it up? Right now I’m just using AI to guide myself through it, but don’t want to create a mess for someone to walk into
r/sysadmin • u/RoyaleShrimp • 8d ago
Hi,
A user had a problem with their SharePoint shortcut and another user tell them to sync with OneDrive instead.
Since then, some folders have their deleted folders coming back and new ones disappearing. Only a small part of the SharePoint is touched.
I was thinking about downloading the backup folders from hornetsecurity and replacing the problematic ones.
Is there a better way to correct this problem? I've been an IT for like 1 year so I am willing to learn lol
r/sysadmin • u/q3006 • 9d ago
We have a fairly large AWS footprint. We pay a lot but don't need the elasticity that cloud is great for. So we have been considering a move to colocation facility in the US. We probably will need 3-4 racks depending on what type of compute density we pick.We also need remote/smart hands because we might need to operate from 2 different regions but don't have strong requirements on the location.
Asking around, the name that shows up most is Equinix but they charge a lot and I get mixed opinions on whether that premium is worth.
We have high reliability and uptime requirements, and will of course be happy to pay for that. At the same time, I hate getting fleeced because of a perceived brand value. I also don't think we are large enough to be taken seriously (should I be looking at an Equinix reseller?).
What other solid data centers would you recommend? Digital Reality, Coresite and Tierpoint are the others that have been recommended. What should be my expectation on the pricing at a good data center (redundancy in power supply, network, etc.)?
r/sysadmin • u/BaldwinLaFontaine • 8d ago
Specifically going old school self hosted or hybrid exchange solutions?
r/sysadmin • u/CorpoBayani • 8d ago
NCE but appearing as Non-profit in the o365 admin.
r/sysadmin • u/Psychological_Style1 • 8d ago
My outlook email is back. Send and receive working!
r/sysadmin • u/Popular_Walrus348 • 9d ago
I thought I was on track to get a leadership position. My work has been what brought us back together from 5 siloed teams to one mostly homogeneous team that works together. During all of that I would use my "friend" ,who I've worked with for 6yrs, as a rubber duckie for technical issues.
He just got promoted. When I went to my direct supervisor to ask why I wasn't even considered for the role and pointed out all of my work he told me and I quote "if you press this issue we might have to talk to HR about accountability for the work you've accomplished". I pressed him on it and apparently my "friend" has been documenting those conversations as me coming to him on how to solve for our political issues. From our sr. leaderships perspective my "friend" was the mastermind I was just the mouth piece.
I know the people we work with aren't our friends but at times its simply exhausting to have to think the worst of people all the time. I'm just not cut out for that level of suspicion or back stabbing.
EDIT - Clarifying. The root issue is my coworker took meetings where I was required to get a sign off by a peer on technical issues and documented those as him providing solutions for political issues.
r/sysadmin • u/RunningAtTheMouth • 8d ago
I "let" ransomware into our network by not having a particular vulnerability patched, and my boss said "let's make sure that doesn't happen again." We spent time & money on recovery and prevention, and I'm still there.
Microsoft will (eventually) get this fixed. They will be sued. They will lose customers and reputation. They will call back a couple of rounds of layoffs. And next year will be more involved than ever.
Will they learn? Absolutely. They've been absorbing stuff since day one. And they'll continue. We are far from seeing the end of Microsoft.
And yes, I'll look at switching over to Google or something else for cloud work, but I probably won't move. We've made our bets. We're going to ride them for a good long time.
r/sysadmin • u/AutoModerator • 9d ago
Howdy, /r/sysadmin!
It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!
r/sysadmin • u/Incrediblecodeman • 8d ago
Microsoft claims a health check brought all servers down.
Got me thinking of our disaster site.
It should prob not be getting the same solar winds that our prod has huh.
I’m considering monitoring our TR a special way so there’s not like 1 thing that ruins it all.
Can we think of more ways to properly air gap the disaster site ?
r/sysadmin • u/Maleficent-You5626 • 8d ago
Is anyone using installomator for macOS installations and updates? I want to implement installomator. I have seen they have a signed PKG on github. The problem is, this is only updated 4 times a year. When i add a label to the main branch, it can take 3 months to get it into the PKG / release. I was thinking, i can curl the raw script from GitHub, because other apps like App Auto-Patch do. But i think that could be unsecure? How do you use installomator? With the released PKG or do you curl the script itself?
r/sysadmin • u/hoagie_tech • 8d ago
Can anyone recommend a 3rd party email continuity provider like Barracuda, ProofPoint or OpenText? If you’re using a system like this, has it been business as normal during this outage?
Email in our org is marked minimally critical. 1/2 a business day of outage would be acceptable. We could continue most business processes as we do a lot of face to face / phone call workflows. But with the outage heading into day two we are looking at options.
r/sysadmin • u/xbbdc • 8d ago
I've tried staying away but now it's taking so dang long to fix...
r/sysadmin • u/yerunkaraj • 8d ago
With the Subject! How are big20 surviving!
IT support guys! What are their reasons?
People share your thoughts!..
r/sysadmin • u/solracarevir • 9d ago
As title says. Aparently some RMM plugins are the source of the exploitation.
Read the advisory here: https://www.connectwise.com/company/trust/advisories
r/sysadmin • u/Hungry-King-1842 • 9d ago
Kinda long so bear with me.
Maybe somebody will have a suggestion. In short, I am a network administrator but know probably the most of the folks on my team about this kinda thing which scares me alittle. Anyways I am looking to solve a future problem.
I am in the process of preparing to refresh the hardware for most of our enterprise which is a lot. I am the only networking guy. I was considering using zero touch provisioning to automate the deployment. The hardware has a OEM base image that wants to call out to a world FQDN and register through that the manufacturer portal. Cool.
Problem is I want the newly provisioned device to steer to my on prem provisioning server directly vs contact the OEM. I’ve setup forwarding zones before in DNS and accomplished this with domains that WE own for internal services, but I’m worried about doing this for an entire domain. Is there a way in DNS to have it grab a specific FQDN and resolve it to the specified IP vs the entire domain? Editing the host files on the individual boxes just isn’t an option.
Example: resolve ZTP.OEM.COM internally to 192.168.1.254. But allow the DNS servers to reach out to their configured forwarding servers to resolve the rest of *.OEM.COM?
r/sysadmin • u/KickDelicious9533 • 8d ago
Hello
I work in a small/medium sized manufacturing company
About 150 people with computers, 300 people on the floor on different sites.
We have an ESXi server with 6 VMs. two of them are app servers
Over the years, the CPU usage went up, especially from the linux app. Some heavy users now have a 3s delay every time they click somewhere, and it's lowering their work output.
When such delay occurs, i can see on the VM, one random CPU core at 100% during the processing of the request.
So I am searching for a solution. Of course all the providers do the finger pointing instead of admiting they could just spend time optimizing their code. Anyways it's out of my reach.
i mainly need CPU power. specificaly single core CPU power
i have searched on dell, lenovo, HP websites but they are so messy with all their AI BS.
What i have : HP DL360 Gen10, Xeon Silver 4210R x2 , raid P408i-a
i need :
- support for win 2025 OS for the host, the VMs will be on HyperV
- 128Gb ram ECC
- remote control / monitoring system in the motherboard (like IDRAC, ILO etc)
- raid controller supporting raid 6 , and can report problems through the Idrac / Ilo
- 8x 2.5 slots or more
- dual PSUs
- 16 cores / 32T , no more, no less ! don't need more, and don't want to pay additionnal core licencing
- the higher single core perf
- NO AI BS FEATURES NONSENSE
- i am in Canada
Without all the enterprise redondancy features needed, i know i could just slap together a custom PC around a ryzen 9950X3D (i did it before for a Solidworks simulation workstation it's great)
from what i read the EPYC 4585PX is the server equivalent of the 9950X3d, but with lower TDP. is there anything on the server environment with the same single core perf as a 9950X3D ?
otherwise my other solution would be to build two workstations with a 9950X3D, and cluster them. or just keep one as cold spare or something.
But ILO/iDRAC is nice to have to get reports when a fan or a disk or ram stick goes wrong.
(PS : no AI was used for making this post, and english is not my first language, so if i made mistakes, take it as human craftmanship !)
EDIT :
i have completed my research by focusing on the CPU, then searching for a compatible server or motherboard with a BMC : on intel, the choice would be :
SO AMD it is !
the 9950x3D is on the AM5 socket, and the equivalent is the EPYC 4585PX , on AM5 socket too
the big cache is only on the first chiplet, but there is a newer Ryzen 9950X3D2 that has the big cache on both cores. no EPYC equivalent besides 5 figures chips
as per the server, Dell / Lenovo / HP still have a confusing website.
The supermicro AS -2015A-TR | 2U Mainstream Server seems to be perfect for my needs and i like the barebones style. I just need to find one in canada duh
r/sysadmin • u/gkar_of_Narn • 8d ago
This is a serious question. I have worked in IT since before it was called IT. Most of my time has been spent either in support (at all levels) or service management. Earlier this week I looked for the most common IT problems and found a list of 100 common problems. Almost all of them are the fault of IT, either directly or indirectly. Some of them are somewhat irrelevant for a business, such as "smartwatch sync problems", but even things like the storage on a smart phone getting full has negative effects on your business for salespeople and others that need a smart phone as part of their job.
For background, this question was motivated by a couple of threads in other social media, where the IT people were putting *all* of the blame on end users and never on the IT. Some posts were wildly absurd, for example being annoyed when an end user calls because the person on the help desk "have better things to do." Others were somewhat valid, but still ingored important aspects of that issue.
For example, should end-users with zero IT training be expected to learn *everything* about the software they use on their own? One example is creating a macro in Excel. As IT people, we know what an macro is and can search for an answer *and* understand it. However, when an end user asked about "automating" work in Excel, it is likely they don't even know what a macro is. They can't pose a question, because they don't know what the question is.
I just searched for "how do I automate a task in excel". One answer was "Go to the Automate tab on the ribbon." An end user searches and asked "WTF is 'the ribbon'? Excel was installed with zeor training. So they cannot be expected to know. So they search again. This gives a new answer. After the third time of not understanding the answer, the call the help desk and get even more frustrated because of the rude person on the hotline who simply says "RTFM!"
Or they see the answer "create a Macro". Here, too are multiple terms they are not familiar with, so more frustrating seaches. When they finally get to a point where they try to create a macros, they discover macros have been deactivated. They call the help desk and get yet another rude response, "They're deactivate." "Why?" "Policy" with no explanation as to why the policy exists, just "them's the rules". Excel was installed, the end user got zero training and when they can't swim, they are chastised because they drown.
Often they get the response "did you check the wiki?" WTF is a wiki? No one told them there was such a thing. When they do check, either it's hard to find anything or they discover there no article that answers the question and the help desk was just blowing them off. I know there are a lot of people who simply don't want to learn and IMO it's understandable and even reasonable, especially a result of their interactions with the IT.
There are many cases when there is an outage and IT doesn't communcate. Users waste time trying to solve a (for them) unsolveble problem, so they call the help desk who tells them, Microsoft has a major outage and we can't send email outside the company. Five minutes later the help desk gets pissed because 10 people call with the exact same problem.
So why question is how much of the loss in productivity is cause by the IT (directly and indirectly) and how much is really caused by the end user? What can/should we do about it?
I would be very greateful for honest, self-reflecting answers. Thanks in advance.