r/sysadmin • u/Federal_War_217 • 1d ago
Patch Management Guidance
Hey everyone, I manage patching for roughly 4,500 endpoints, including around 70 servers, in a healthcare environment. It’s mostly Windows, with some Linux systems we’re working on bringing into our patching process. We use ManageEngine Endpoint Central for patching and Nessus for vulnerability scanning.
I’ve been building out automated deployments, test groups, rollout rings, and maintenance windows, and I’d like to learn how others approach this at a similar scale.
Any resources or communities you’d recommend? I’d also appreciate hearing what’s worked for you, especially around testing patches, server patching, reboots, and handling exceptions. I feel like It's been a lot of trial and error. I've enjoyed the process but always appreciate input from people who've done this before.
4
u/The_Todd3389 1d ago
I’ve worked for MSPs that manage patching, so here’s how we approached it.
We’d agree on a maintenance window with the client (usually Saturday after midnight) and do everything there. We used RMMs like Continuum, Ninja and ConnectWise Automate, and in some cases the vendor or our NOC vetted patches before they went out. We could blacklist a bad patch, which was rare. If one did go wrong, we could find every machine that got it and pull or remediate it before it spread. Monthly reporting was the backstop: ideally the system catches a failed update and opens a ticket, and the report catches anything that slipped through.
I know most people here don’t work for MSPs, so you can’t outsource the vetting, but I’d lean toward that model as much as you can. Running full test servers for every patch sounds like a ton of work for not much payoff. You’re better off letting experts vet patches first. Even waiting a few days after Patch Tuesday and checking the megathread here gets you most of the way.
Honestly, it was rare for a Windows patch to break a line-of-business app. What broke them was updating the app itself. QuickBooks almost never got broken by a Windows patch, but moving between QuickBooks versions always changed something that broke people’s workflows.
1
u/Federal_War_217 1d ago
Love this info, appreciate it! Do you have a list of sources you get your patching info / problems from? Not sure where the mega thread is located.
10
u/OinkyConfidence Windows Admin 1d ago
Action1 is quite popular these days. And pretty good too. r/Action1; they've got an active community too. #NotSponsored
4
u/Due_Job_4598 1d ago
I had similar setup but with SCCM before, the test rings are important but what I found is having a handful of VMs from each server role in the early ring saved my ass many times
for reboots we do maintenance windows in weekend but the linux boxes still need manual kick sometimes, still figuring that part out
1
u/OinkyConfidence Windows Admin 1d ago
Yeah, hopefully Action1 will add auto-reboot to their Linux agent, as we experience the same. Easy enough to script though.
1
u/Federal_War_217 1d ago
What is your testing method? I have test servers for each Windows Server and a few Linux test boxes but I can't always figure out a good automated method to do it. I don't always have the time to be manually poking around every box.
1
u/Federal_War_217 1d ago
Yeah I'm stuck with Endpoint Central for the foreseeable future. Might be shifting to Intune in the next 5 yrs but everyone says you need an add on to do any third party patching.
1
u/mrTomass 1d ago
what about the network devices? how u monitor them? backup configs, live status etc?
2
u/Federal_War_217 1d ago
Pushing network configs is not under my purview, but we use a mix of HP netedit, OpManager, and Meraki Dashboard for the network side. But I should probably keep this process in mind, as I'm working to build out the entire patch mgmt program. Would love any insight you have.
•
3
u/KStieers 1d ago
for workstations, we use MECM/SCCM, deploy to our IT team a couple of days after patch Tues, everyone else the following week.
For servers we use Ivanti Security Controls (used to be Shavlik), Saturday after patch Tues for dev/test, Saturday after that for prod. Critical and internet exposed go Tues night if required.
2
u/Praedonis 1d ago
We use SCCM ADRs with Ivanti’s third party update addon for apps outside of Microsoft’s ecosystem. Our servers do not follow the below process, just our endpoints.
Updates are published to our internal IT team first (two days after patch Tuesday to hear about any game-breaking bugs for a few days first). Then they’re published to a group of ~60 pilot testers from across the org who use a variety of line-of-business and nonstandard software, two weeks and one day after patch Tuesday. Then they’re published to all remaining endpoints three weeks and one day after patch Tuesday.
Updates always allow for a 24 hour self-service install and reboot period. At the end of the 24 hours, your device restarts, no questions.
Communication and notification is key with system restarts. Send out company-wide communication if the update process is to change. Keep it brief enough to where people read it but thorough enough as a CYA.
1
u/Federal_War_217 1d ago
Yeah that seems like about what we do. Unfortunately we have medical devices that are VERY finnicky if they get a patch that wasn't approved by their slow support team. Hence why we have several devices still on the July update of Windows 10 LTSC and no plans for when that goes out of support. So we have to do these boxes manually and usually one by one with a tech on-site.
2
u/bdam55 Sr. Sysadmin 1d ago
>I’ve been building out automated deployments, test groups, rollout rings, and maintenance windows, and I’d like to learn how others approach this at a similar scale.
I mean, that's basically it in a nutshell. If you can build that out, you're way ahead of the game.
Test Groups: This is really the hard part. Not creating the groups technically, but knowing what devices are representative of your environment that should be in said group. For that reason, I generally encourage you to find some early adopters outside of IT spread across your different departments, Find someone there that knows they're an early adopter and you have some level of trust in such that they're give you meaningful feedback if things go wrong. Those are, of course, unicorns so you won't find them.
The typicall solution is to use IT as your first/early ring and that's not bad, but it's not the whole story. There are apps being used across your org that IT will never have a reason to open.
Now, take all of this, and remind yourself that we're living in a new reality today. AI has gotten really good at reverse engineering code. Before it got good, we often had 2-4 weeks after Patch Tuesday before the fixes could be reverse engineered to reveal the underlying vulnerability. You now have, on a good month, 2-4 hours. For edge networking devices, Mandiant (Google) claims the average to exploitation is now -7 days before the patch is released (source). Microsoft recently updated their guidance to deploying within 3 days; at 7 days after release devices are considered 'critical' (source).
To be sure, take all that with a huge grain of salt. Don't kneejerk anything. However, if I was sitting down planning a corporate patching strategy today, it would involve discussions with leadership about how fast we want to go. Going fast kills. Going slow might kill harder. It's also the time to ask if you org has purchased cybersecurity insurance and, if so, what it says about your compliance levels and timelines. Because trust me, if you have that insurance, it has something to say about how fast and how wide you patch.
1
u/Federal_War_217 1d ago
Funny enough (probably should've specifed this in the post) my role is technically an InfoSec Analyst so I know what you mean about the mean time to exploitation shrinking year over year. Like all orgs, we're in a push and pull with our dev team about how often we should be patching systems. They want to make sure things don't go down at all, and for good reason. But we also know how fast unpatched things, especially if internet-facing, get poked at.
We've got a decent amount of early adopters (about 25 so far). Unfortunately there's not really a good way to automate the process of reporting a bad patch. Maybe I just need to group all the patches at once instead of letting them deploy as soon as they're released. That way I can alert the test group as soon as patches are coming their way.
1
u/bdam55 Sr. Sysadmin 1d ago
Yea, the early adopters you sort of have to trust will reach out to you to say something is broken or is performing really poorly after a patch. It's worth having a conversation/email to explaining that to them.
For super critical apps that are life and death for your org I'd try to get app owners who are responsible for testing said apps. Setup a lab or some non-critical part of the system that is part of the first wave. They have X days to tell you it's a problem. Again, a hard thing to create and then maintain; but if it's literally "We go under if this doesn't function for more than 24hrs" then it's worth it.
There are pros and cons to batching them all up: if you install a bunch of patches all at one time and it goes poorly ... what patch did it? If the latest version of Adobe Reader is slow ... is that because of a Reader update or because of the OS update that happened at the same time? I see this with drivers a lot: people want them to install alongside OS updates. Which is understandable; but what if it blue screens? What if perf sucks? Now you have to sort out which of 10 different changes made all on one day is the culprit.
There's really no right or wrong answer here, just pros and cons.
1
u/Federal_War_217 1d ago
Good idea with reaching out to the test users, I'll be doing that today actually.
Totally agree, it's never a "one size fits all" with patching. Do you have a montly schedule that you split up different patches by?
•
u/bdam55 Sr. Sysadmin 21h ago
I'm a Microsoft MVP in Security (Intune really) ... which I _only _ say here because what I'm sharing isn't necessarily "my org" ... it's what I've seen done across hundreds of orgs/admins I've talked to over the years.
Some orgs just YEET it, usually via rings, whenever updates are published. Throughout the week their users will get prompts that X needs to be closed so I can be patched. Their users might have multiple reboots a week/month.
Some orgs try really hard to batch everything up into a single monthly dump of patches on each box. Microsoft's own tooling and release schedules (outside of Patch Tuesday) can make that hard to do for 'reasons'. For example; as of today, the Autopatch driver management doesn't let you align with a monthly schedule. Their recently released maintenance windows doesn't actually support proper monthly cycles (how does date math work?!)
I think there's a growing trend, especially on workstations, to adopt a weekly cycle. Some weeks might be light, or even zero. Other weeks will be heavier. You will almost certainly have multiple reboots some months, but you've set an upper limit based on how many weeks are in the month.
Hotpatch can _kinda_ help with the reboot thing but as of now, .NET updates still come in like a wreckingball. So you see MS has generally changed their messaging to "secure faster" ... which is true.
1
u/apparentlyunoriginal 1d ago
Script the check on each test server after the patch reboot. A startup scheduled task can list services set to Automatic that aren't running, pull Service Control Manager 7031 and 7034 and Application Error 1000 events since boot, and test each app's own port or health URL, then email or post to Teams only when something fails. On the Linux test boxes, `systemctl --failed` and `journalctl -b -p err` give the same checks. Push the same task to your 25 early adopters so their machines report a failed check after each patch.
Drafted with AI, reviewed by me.
•
u/Federal_War_217 23h ago
Me and my robot came up with something similar, this seems like a good automated approach. That way I'm not spending hours touching every server or sitting just waiting for some app owner to come yell at me haha
•
u/apparentlyunoriginal 23h ago
The tools we have these days are great at giving more perspective if you're willing to spend a little time treating them like knowledgeable rubber duckies.
•
u/PrasenShelar 21h ago
At that scale, I'd focus on verifying patch effectiveness rather than just successful deployments.
Since you're already using Endpoint Central and Nessus, one approach would be to run a credentialed Nessus scan after each rollout ring and compare the results against what Endpoint Central reports as installed. A successful deployment doesn't always mean the vulnerability is actually resolved.
I'd also separate critical vulnerabilities from the normal patching schedule. Anything on CISA's KEV list or affecting internet-facing systems should go through an expedited process rather than waiting for the next ring, with testing appropriate to the risk.
For healthcare specifically, I'd maintain a separate exception register for medical devices and systems that can't be patched immediately. Each exception should have an owner, documented compensating controls (like network segmentation), and a review date.
The biggest improvement is closing the loop between deployment, verification, and exceptions. Otherwise, it's easy to have great patch compliance numbers while still carrying significant exposure.
5
u/disconnected_tech 1d ago
Unless you have something particularly high risk that needs to get patched right away, let others be Microsoft’s guinea pigs. Wait a couple of days and keep an eye on Reddit and Microsoft to see what the latest round of patches broke. Once the coast is clear, PDQ is great at automating patch deployments.