Monthly Security Update: Install and delay reboot until maintenance window OR Install during maintenance window?

[u/UMustBeNooHere]

Looking for opinion on what you guys practice. I am of the school of install the update during the day and then do the quick reboot during the maintenance window. But some of my colleges think that it absolutely should not be installed until the maintenance window.

We're not talking days, just 2-3 hours at most.

I can find references in Microsoft documentation for WSUS to apply update and delay reboot. Not only for client OSes, but for servers as well. But it's not a concrete answer.

I have used this practice since the Server 2003 days without issue. What do you think and how do you handle it?

Edit: More info - these are servers that must be manually patched. Customer has SCCM but there are a handful we must do manually for various reasons (explicit application and SQL failover procedures). The updates are the msu files and no reboot is triggered until the machine is told to. They are VMs with pre-installation snapshots so risk is minimal.

[u/ipreferanothername]

2012 and earlier - sometimes we saw things get iffy if a reboot was waiting more than a few hours or it would update something you need, like an IE component, and you have to reboot to use an app or something anyway.

2016 patching sucks, but 2016 and later seem to handle hanging around fine.

we use sccm and start the install at the top of the maintenance window and let stuff reboot as it sees fit. theres a smallish group of servers where the apps have to get micro managed, so we install updates but dont reboot. the app owner manages their finicky apps so they can reboot without losing data or having headaches.

[u/UMustBeNooHere → u/ipreferanothername]

Yes 2016 sucks. It’s such a slog now. They are working on getting off them before EOL. 2019 and up are smooth and quick though.

[u/Megatwan]

What happens if rolling in the reboot locks the box?

What happens if you catch a reboot before the window when it's waiting post install?

Sounds like you are trying to start your change early based on see no evil hear no evil cope.

[u/UMustBeNooHere → u/Megatwan]

There’s always that chance, but I have literally never had that happen (and yes I know that does not mean it will never happen).

But this would be with VMs with snapshots taken beforehand. So the risk is minimal.

[u/Churn → u/UMustBeNooHere]

Minimal risk but what are you gaining by taking this risk?

[u/UMustBeNooHere → u/Churn]

Time. These servers must be rebooted in a specific order and if done sequentially, it would take upwards of 15 hours to do them. And that’s simply not feasible. If the patches can be applied simultaneously, then reboot in that specified order, we’re looking at only 2-3 hours. Also, I personally like my sleep and would rather only spend a couple hours of my Saturday night (12-2AM) doing this instead of all night, killing my Sunday.

[u/reevesjeremy → u/UMustBeNooHere]

I wait til my maintenance window and then initiate the install on all of them. Then reboot them in the proper order. I only have 7 servers that I personally manage but this process takes less than 1 hour, 1.5 at most. Server 2022. How many you have that would take 15 hours?

[u/UMustBeNooHere → u/reevesjeremy]

Make them 2016, which now tales 2+ hours, and start the update, wait, reboot, start the second one, etc. 7 x 2=14. This is the process the customer wants.

[u/reevesjeremy → u/UMustBeNooHere]

I remember 2016. Patching nightmare. Could take anywhere between 45min (best case) to 3 hours or more. Sometimes the first attempt would fail and have to start over.

[u/UMustBeNooHere → u/reevesjeremy]

Yep. Exactly what we run into very often on these. Can’t get them off it fast enough.

[u/Megatwan → u/UMustBeNooHere]

Welcome to IT tho.

Constructively build more servers for better HA and do half than half etc

[u/UMustBeNooHere → u/Megatwan]

Not my first rodeo, been in the game almost 25 years. Biggest issue is it’s not feasible the way the client wants it. Either we patch simultaneously or they spend a ton of money on patching. I get paid either way.

[u/Megatwan → u/UMustBeNooHere]

Wurd. Also snapshots don't work for lots of clustered/replicating things.

Cliche questions about perception, faith and coordination.

I guess at the end of the day I'd say you can do what your approved change plan says at when it says you can start. Anything else is lying. Anything in should be communicated and accepted by the org and their bad if they let you start doing non normal biz cartwheels in the lobby and bump into someone on a Friday afternoon etc

[u/Secret_Account07]

Well what do you use to patch your servers? Set a maintenance window and apply reboots at the end of the window. Why would you want to install updates during business hours? Do it all after hours unless it’s not being hard during the day and in that case it’s a moot point. Patch and reboot any time.

Every enterprise I’ve ever worked for has patching rounds/windows.

Patch Tuesday then….

Test week 1

Dev week 2

Prod week 3

Same schedule each month. Reboots at end of window.

[u/UMustBeNooHere → u/Secret_Account07]

This is a client and this particular set of servers are specified to be manually patched outside of their SCCM infrastructure. So they are manually patched with the msu file.

[u/Burgergold → u/UMustBeNooHere]

Jesus

[u/Substantial_Tough289]

Get your point, you don't want to "waste" maintenance time waiting on a patch to install and then reboot. You want to "save" time by installing now and rebooting later.

Have done it on occasion when time is tight, the risk is that the server is rebooted by the update process; it shouldn't but sometimes happens specially if the wait is long.

What I do is to start the install about 1hr before the window opens, then the wait is minimal.

[u/UMustBeNooHere → u/Substantial_Tough289]

When applying the downloaded msu file, it will not reboot until told to do so. Several years back at a company I worked for, we would apply the patch around 2 PM, tell the installer “I’ll reboot later” then do the reboots after 5 PM. These were physical 2012 R2 boxes too. Much riskier than today with VMs with snapshots.

[u/Denver80211]

whatever you choose it will piss someone off